matrixmultiply

matrixmultiply does f32 and f64 matrix multiplication quite competently for larger matrices. It's a simpler dependency than blas, so it might be nice to just use. It does need type specific dispatch (unstable stabilization, TypeId, or specific trait) at this point though.
What do you think, could I help you integrate with it, or are you developing something equivalent?

Hey!
Thanks for creating this issue. I was planning on reaching out to you soon to ask about it too.
I think it would be really good to use the matrixmultiply library! I had started playing around with some divide and conquer parallelism and after wrapping up my current work was going to compare the two. I'd definitely like to compare how big the gains are compared to my current (in-the-works) parallelized approach.
Do you have plans to add multithreading to the library - I'd be happy to try and help with that!

I didn't have any plans for that, even if the shape of it should be set up for it. I've already tried plugging in rayon to try it though. It seems it needs quite big matrices for it to pay off.

Hmm, I found that I had some nice performance gains even for matrices that were as small as n x 64. I imagine even without parallelism your library will achieve higher performance. I'd be happy to play around with parallelism a little in matrixmultiply but for now I'll experiment with it as is.
Before starting any work I need to break up the matrix module into smaller sub modules (all the operation overload etc. is in one file right now, and without macros!). I'll definitely need some help integrating it and will keep this issue open.

The packed matrix multiply will already be many times faster than a simpler approach though, already at small sizes. (This comparison for example https://github.com/bluss/matrixmultiply/issues/6#issuecomment-205079608 The "ref" version is a reference simple triple loop matrix multiplication implementation).

For tracking:
I'm pretty convinced that swapping out the current multiplication methods for matrixmultiply is an easy and smart move. As for multithreading I can experiment with divide and conquer which calls matrixmultiply at the base level.

Did a quick test using matrixmultiply:
Before:
test linalg::matrix::mat_mul_128_1000       ... bench:  14,231,183 ns/iter (+/- 833,129)
After:
test linalg::matrix::mat_mul_128_1000       ... bench:   3,352,754 ns/iter (+/- 412,408)
Pretty nice! This is without any divide and conquer parallelism as well (which I will need to investigate).
The only thing holding me back now is that my current multiplication is generic while matrixmultiply supports only f32 and f64. To solve this for now we could use:
http://rust-num.github.io/num/num/traits/trait.ToPrimitive.html
But I still need to cover both f32 and f64. @bluss I guess this was what you were talking about with specific dispatch. Do you have any suggestions for handling this?

I'm not surprised it was a big improvement :wink:
There's no perfect solution to type specific dispatch without specialization (that young unstable feature). ndarray uses Any and TypeId to use the f32, f64 for those types only, but it requires a trait bound Any. You can also define your own trait, but it would not be as inclusive as the current trait bounds.
I have experimented some with scoped_threadpool and rayon in matrixmultiply, maybe not with big enough matrices. It's easy to have overhead dominate. I think the sweet spot to divide by multiple threads are the loops marked 3 and 2 in that code, which requires some extra coding since each thread then needs its own packing buffer.
Oh by the way, there's another alternative; to write a more generic version of matrixmultiply. It's not my favourite solution for several reasons including compilation time, the matrix multiply then needs to be recompiled every time the usage site of it is recompiled.

I was going to see if I could get away with simply dividing up the matrix before feeding it to the BLIS algorithm. If that doesn't work I'll try to play with the algorithm itself - but the gains are big enough that I'm not super motivated to improve it :).
I think it is best not to have a more generic version matrixmultiply if it impacts compilation heavily.
I'll take a look at how you solve the dispatch in ndarray - no point reinventing the wheel ;). I haven't played with Any or TypeId yet so that may be fun...
Thanks

I call the TypeId solution "ad-hoc" specialization. It's crude but it works just fine.

Dividing up the matrix sounds just fine actually.

(Just make sure the threads are not writing to the same output location in the C matrix)

Source code https://github.com/bluss/rust-ndarray/blob/master/src/linalg/impl_linalg.rs#L402-L473

Thanks for the source - that doesn't look too bad at all :).
It could be a challenge to get the divide and conquer right - as you pointed out, providing the right views of C to dgemm will take some careful thought.

Pen and paper solves everything :wink:

A quick and dirty implementation shows we can get some speedup:
test linalg::matrix::mat_mul_128_1000       ... bench:   3,154,849 ns/iter (+/- 243,448)
test linalg::matrix::mat_paramul_128_1000   ... bench:   2,308,833 ns/iter (+/- 970,111)
This is splitting along an axis with length greater than 256 (I tried 64 and 128 as well). This implementation also has huge overhead as I'm copying data to split up the matrices (I need to rewrite it using MatrixSlice). I'm also just creating new C matrices for each actual dgemm call. This can also be improved :).

Cool. So you're the parallelism wizard

Which crate do you use for threading by the way?

I'm not sure about wizard! It is a simple but inefficient implementation. I'm using rayon for the threading, it looks like this:
pub fn paramul(&self, m: &Matrix<T>) -> Matrix<T> {
        let n = self.rows();
        let p = self.cols();
        let q = m.cols();

        let mut max_dim = n;

        if max_dim < p {
            max_dim = p;
        }
        if max_dim < q {
            max_dim = q;
        }

        if max_dim < 256 {
            // If max_dim is less than some threshhold, just multiply.
            self * m
        } else {
            // Otherwise we should split along the axis.
            let split_point = max_dim / 2;

            // Split along `self`'s rows.
            if max_dim == n {
                let top = self.select_rows(&(0..split_point).collect::<Vec<usize>>()[..]);
                let bottom = self.select_rows(&(split_point..n).collect::<Vec<usize>>()[..]);

                let (a_1_b, a_2_b) = rayon::join(|| Matrix::paramul(&top, m),
                                                 || Matrix::paramul(&bottom, m));

                a_1_b.vcat(&a_2_b)
            } else if max_dim == p {
                // Split self vertically and b horizontally
                let a_left = self.select_cols(&(0..split_point).collect::<Vec<usize>>()[..]);
                let a_right = self.select_cols(&(split_point..p).collect::<Vec<usize>>()[..]);

                let b_top = m.select_rows(&(0..split_point).collect::<Vec<usize>>()[..]);
                let b_bottom = m.select_rows(&(split_point..p).collect::<Vec<usize>>()[..]);

                let (a_1_b_1, a_2_b_2) = rayon::join(|| Matrix::paramul(&a_left, &b_top),
                                                     || Matrix::paramul(&a_right, &b_bottom));

                a_1_b_1 + a_2_b_2
            } else if max_dim == q {
                // Split m vertically
                let left = m.select_cols(&(0..split_point).collect::<Vec<usize>>()[..]);
                let right = m.select_cols(&(split_point..q).collect::<Vec<usize>>()[..]);

                let (a_b_1, a_b_2) = rayon::join(|| Matrix::paramul(self, &left),
                                                 || Matrix::paramul(self, &right));

                a_b_1.hcat(&a_b_2)
            } else {
                panic!("Couldn't find the max of the matrix dimensions.");
            }
        }
    }

The select methods are copying data right now. I need to change them to use split methods and get slices.
The basic idea of the algorithm is described here. Split in half along the largest axis and then bring the results back together.

Nice, so there's some easy wins possible by using views for the inputs to the multiplication.

I suspect so :).
Sadly I need to do some code maintenance before I can get to work on this properly. I'll probably poke you at some point for some advice with that too...
Once I've wrapped that up I'll try to get this going. It should make the performance of most algorithms pretty acceptable!


Disk and SWAP space on MinetestForFun server

#### Hard Drive Disk

[x] The server is almost out of space, i need to add a 20GB more space for reach a 40GB of total space !
Add a more larger HDD is easy and jsut need 2 clicks, the problem is for the re-partitionning OS side of the system, i need to reboot the server with a Live Gparted OS, and move partitions needs times... (40 to 50 MB/s of writing on my dédibox)

SWAP

[x] Same has the Hard Drive disk, the SWAP is added since a long time, but not tweaked with GPARTED live OS, it's a mistake since a very long time :/ (don't worry, the SWAP is nearly never used), i will add a 8GB of SWAP space soon
[x] Add ZSWAP/ZRAM from https://wiki.debian.org/ZRam

Everything is done right now, closed the issue

Re-opened, the ZRAM needs to have a more high priority than SWAP
Actually SWAP is -1 and ZRAM is ???

The Debian script swapons zram0 with a priority of 10, so it's good already.

Great ! Thank you for this comment :)


ABOUT GPU RAM

I run the playground.py file with 16GB of GPU ram but it's still not enough. Can you show me some ways to still run files but without needing 16GB of GPU ram?

Similar problem, I run the playground.py with 24GB GPU RAM, it still gives torch.cuda.OutOfMemoryError.

Thanks for your attention to this work. I have updated the playground.py for model loading so that 24GB VRAM should be available to run, but I have no idea how to make it run on 16GB VRAM for now. I'm sorry about that.

Thank you for your response


make docker push fails on OSX

Running this command...
BUILD_WITH_CONTAINER=1 make docker push

Yields this output:
/usr/local/bin/docker-entrypoint: line 57: exec: gmake: not found
gmake: *** [Makefile:46: push] Error 127

I have the docker desktop installed.

@sdake IIRC you were looking into build issues on OSX ... should we include this as well?

This is a new one. Normally it's a permission issue with the solution to follow the wiki.

@nmittler The build tools container was working well on Mac in the past. The makefiles were rewritten into scripts, and I approved the PR, so I am fixing the fallout.
The build container itself doesn't build. I don't know if there is an issue for this, but I nearly have a solution.
Once that is fixed, the problem you are running into is a simple path issue and relatively easy to fix.
Make docker push may also be broken (just on Mac) - which would be the third issue :)

A work-around I've found for now might be to prepend your command with a MAKE=make as well.

If the work-around works, or please try this again now that https://github.com/istio/common-files/pull/223 is merged.

BUILD_WITH_CONTAINER=1 make docker push
...
mkdir: cannot create directory '/Users': Permission denied
mkdir: cannot create directory '/Users': Permission denied
make[1]: *** No rule to make target '/Users/nathanmittler/go/out/darwin_amd64/release', needed by '/Users/nathanmittler/go/out/darwin_amd64/release/istio_is_init'.  Stop.
gmake: *** [Makefile:46: push] Error 2

You don't need the BUILD_WITH_CONTAINER=1.  And I would have thought make docker.push Let me try this locally. Did you try make clean to see if that works?

I agree there is an issue with things currently. As a work-around, I think one can do a make shell to get to a command line in the container and the do the make docker.push from there. At least that's working for me locally. Will dig deeper into what's causing things to fail when run from the host command line.

Just tried without BUILD_WITH_CONTAINER:
make docker push
...
fpm -s dir -t deb -n istio-sidecar -p /Users/nathanmittler/go/src/istio.io/istio/out/linux_amd64/release/istio-sidecar.deb --version 1.6dev -f \
	--url http://istio.io  \
	--license Apache \
	--vendor istio.io \
	--maintainer<EMAIL_ADDRESS>\
	--after-install tools/packaging/deb/postinst.sh \
	--config-files /var/lib/istio/envoy/envoy_bootstrap_tmpl.json \
	--config-files /var/lib/istio/envoy/sidecar.env \
	--description "Istio Sidecar" \
	--depends iproute2 \
	--depends iptables \
	 /Users/nathanmittler/go/src/istio.io/istio/out/linux_amd64/envoy=/usr/local/bin/envoy /Users/nathanmittler/go/src/istio.io/istio/out/linux_amd64/pilot-agent=/usr/local/bin/pilot-agent /Users/nathanmittler/go/src/istio.io/istio/out/linux_amd64/node_agent=/usr/local/bin/node_agent /Users/nathanmittler/go/src/istio.io/istio/tools/packaging/common/istio-start.sh=/usr/local/bin/istio-start.sh /Users/nathanmittler/go/src/istio.io/istio/tools/packaging/common/istio.service=/lib/systemd/system/istio.service /Users/nathanmittler/go/src/istio.io/istio/tools/packaging/common/sidecar.env=/var/lib/istio/envoy/sidecar.env /Users/nathanmittler/go/src/istio.io/istio/tools/packaging/common/envoy_bootstrap_v2.json=/var/lib/istio/envoy/envoy_bootstrap_tmpl.json
/bin/bash: /usr/local/bin/fpm: /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/bin/ruby: bad interpreter: No such file or directory
gmake: *** [tools/packaging/deb/istio.mk:50: deb/fpm] Error 126

And make shell doesn't seem to exist as a target:
make shell
gmake: *** No rule to make target 'shell'.  Stop.

Looks like I had BUILD_WITH_CONTAINER=0 set in my environment. Needed to remove/override that to get shell to work. make docker push seems to be progressing further in the shell.

https://github.com/istio/istio/pull/22700 was for a problem where there was recursion and a failure message when trying to use make docker or make istioctl:
make docker
/Users/ericvn/work/go/src/istio.io/istio/common/scripts/setup_env.sh: fork: Resource temporarily unavailable
/bin/bash: fork: Resource temporarily unavailable
/Users/ericvn/work/go/src/istio.io/istio/common/scripts/setup_env.sh: fork: Resource temporarily unavailable
make[2395]: `docker' is up to date.

With that change, I am able to do a make docker (which defaults to BUILD_WITH_CONTAINER=1 now), and it runs in the container successfully and a subsequent push worked as well (one does need to make sure they follow the wiki to set up the Mac Docker creds successfully).

Hello @nmittler. Can you confirm how you are running that command? From the command line locally, as part of a target on a make using the container default?
You may see a problem inside the container if you do not have your Docker for Mac setup as per the wiki: https://github.com/istio/istio/wiki/Preparing-for-Development-Mac
There is a problem when using docker commands in the build container if things are setup as per the wiki. Mariam and Steve had come up with that way to get the credentials inside the container. I did have a command that can be used to get the creds if you store them inside the KeyChain, and there was discussion of using that at some point, but I think that work stalled. I might be able to look at it again as a quick fix.
Personally, the wiki doesn't work for me as I can't unselect the option. So I have to create the ~/.docker/plaintext-passwords.json manually. I had thought about adding those directions to the wiki, but didn't since the other change was supposedly imminent.
Let's confirm that this is indeed an issue with getting the creds from within the build container and then we can come up with an action plan.

I'm going to close this issue for now. I was able to build and push the latest istio master this morning.
Please reopen if you still have issues and/or think we need to attach this problem with a wiki or code update.
Thanks!

@ericvn I am having similar trouble. What do you have in the ~/.docker/plaintext-passwords.json? I would like to create it manually so that I can continue building?
Thanks

I figured out how to disable Securely store Docker logins in macOS keychain https://github.com/docker/for-mac/issues/3774#issuecomment-612586200
Thanks

Just getting back to this ... looks like my main issue was that I had ISTIO_OUT set, which was just generally causing some bad things to happen in the makefile. Looks like things are working now.  Thanks for the help!


Bugfix agent config

https://github.com/rancher/dashboard/issues/9003
In addition to the issue above this PR also fixes an issue where switching an affinity term between from required to preferred clears out other required terms, and some instances in which the match expressions/field select appeared in pod affinity terms (which should only be available in node affinity terms)

I'm not sure if I quite understand the reasoning behind replacing labelSelector with isPod throughout this change. My assumption at this point is that labelSelector was being used incorrectly throughout this point.

Yes - using the entire labelSelector object in the match expressions component in the pod affinity context and only the match expression array in the node affinity context introduced complexity with no real benefit, but the component still needs to track whether its being used in the pod or node affinity context as the operator options are different, hence, the isPod boolean


verify(): split out method to retrieve current keys from actual verification

The initial version of verify() is based on @jasonetco's verification code for the @models app: https://github.com/copilot-extensions/github-models-extension/blob/46a3c66f6e6b11a174c1eff0440914a7a211cf2a/src/validate-signature.ts#L35-L66
What I don't like with this implementation is that it does two things

Retrieves the currently valid keys
Validates the payload

This makes caching the response of valid keys unnecessarily hard. We should split it out into two methods and then consider providing a higher level method which combines both for convenience, if that's what the user wants.

done via #8


Openshift issue

Hello,
Your library works perfectly locally but when used in openshift, I encounter trouble.
I receive this error. Do you know why ? Is it possible to configure your package to use specific ports ?
events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: listen EACCES
    at errnoException (net.js:905:11)
    at Server._listen2 (net.js:1024:19)
    at listen (net.js:1065:10)
    at net.js:1147:9
    at dns.js:72:18
    at process._tickCallback (node.js:442:13)

I've read this interresting article in which they say that phantom generates a little local server bind to localhost. Openshift does not allow that.
I was wondering if there is a way to initialiez your package with another variable that localhost ?
http://stackoverflow.com/questions/32260650/create-a-pdf-from-html-and-send-it-to-the-user-from-a-buffer/34039207
I have tried initializing the library using this :
var conversion = require("phantom-html-to-pdf")({
  host: process.env.OPENSHIFT_NODEJS_IP,
  phantomPath: require('path').dirname(process.env.PHANTOMJS_EXECUTABLE) + '/'
});

But his does not work.
Using the library with this :
 var conversion = require("phantom-html-to-pdf")({
    strategy: "dedicated-process"
});

Is working locally but not in openshift.
In openshift, I receive a "failed to load pdf error" from my browser but no error server side ...

Hi,
please try this:
var servHost = process.env.OPENSHIFT_NODEJS_IP || '<IP_ADDRESS>';

var conversion = require("phantom-html-to-pdf")({
    host: servHost,
    portLeftBoundary: 15000,
    portRightBoundary: 35530
});

OpenShift allows to start phantomjs server only in specific port range. That is exactly why we have these options.

I've tried. Locally it works.
In openshift however, I'm still having trouble. No data come in the callback of the conversion call ...
conversion(
            {
                html: html,
                paperSize: {
                    format: 'A4',
                    orientation: 'portrait',
                    margin: '1cm'
                }
            },
            function(err, pdf) {

                if (err){
                    console.log(err);
                    def.reject();
               }
                else{
                def.resolve( pdf.stream );
                });

/var/lib/openshift/56a5028b2d52716b390000e2/app-root/runtime/repo/website/tools/utils.js:79
                def.resolve( pdf.stream );
                                ^
TypeError: Cannot read property 'stream' of undefined

This works for me as the first try....
var server_port = process.env.OPENSHIFT_NODEJS_PORT || 8080
var server_ip_address = process.env.OPENSHIFT_NODEJS_IP || '<IP_ADDRESS>'

var http = require('http');

var server = http.createServer(function(req, res) {
    var conversion = require("phantom-html-to-pdf")({ strategy: "dedicated-process"});
    conversion({ html: "
Hello World
" }, function(err, pdf) {
        res.setHeader('Content-Type', 'application/pdf');
        pdf.stream.pipe(res);
    });
});

server.listen(server_port, server_ip_address, function () {
  console.log( "Listening on " + server_ip_address + ", server_port " + server_port)
});

And with server strategy also
var server_port = process.env.OPENSHIFT_NODEJS_PORT || 8080
var server_ip_address = process.env.OPENSHIFT_NODEJS_IP || '<IP_ADDRESS>'

var http = require('http');

var server = http.createServer(function(req, res) {
    var conversion = require("phantom-html-to-pdf")({
        host: server_ip_address,
        portLeftBoundary: 15000,
        portRightBoundary: 35530 });
    conversion({ html: "
Hello World
" }, function(err, pdf) {
        res.setHeader('Content-Type', 'application/pdf');
        pdf.stream.pipe(res);
    });
});

server.listen(server_port, server_ip_address, function () {
  console.log( "Listening on " + server_ip_address + ", server_port " + server_port)
});

http://test-pofider.rhcloud.com/

Well, I just don't get it. My code is the same as yours and it works
locally ...
Seems that openshift is still blocking something ...
On Tue, May 3, 2016 at 2:02 PM, Jan Blaha<EMAIL_ADDRESS>wrote:

http://test-pofider.rhcloud.com/
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/pofider/phantom-html-to-pdf/issues/30#issuecomment-216507633

--
Jean Chapelle

You should probably try to start over from the scratch.
I've created just plain nodejs-0.10 application and added the mentioned code to the server.js

Ok, I've found the problem.
The problem came from an image url inside my html that wasn't properly
resolved.
The strange thing is that an error was generated my your conversion method
but my promise.reject didn't interrupt the flow ...
On Tue, May 3, 2016 at 4:02 PM, Jean Chapelle<EMAIL_ADDRESS>wrote:

Well, I just don't get it. My code is the same as yours and it works
locally ...
Seems that openshift is still blocking something ...
On Tue, May 3, 2016 at 2:02 PM, Jan Blaha<EMAIL_ADDRESS>wrote:

http://test-pofider.rhcloud.com/
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/pofider/phantom-html-to-pdf/issues/30#issuecomment-216507633

--
Jean Chapelle

--
Jean Chapelle

I guess we can close this.

Off course, thanks a lot for your support
On Tue, May 3, 2016 at 4:41 PM, Jan Blaha<EMAIL_ADDRESS>wrote:

I guess we can close this.
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/pofider/phantom-html-to-pdf/issues/30#issuecomment-216550455

--
Jean Chapelle


sharp(new Buffer(svg)) throws "corrupt headers" error whenever svg is larger than 10,000,000 bytes

Firstly, thank you for providing this robust and useful library!
We are encountering some difficulty with this piece of code:
exports.svgToPng = function(svg) {
   return new Promise(async function(resolve, reject) {
   let buffer = Buffer.from(svg);
   console.log(buffer);
   await sharp(buffer)
      .png()
      .toBuffer()
      .then(data => {
         console.log('svgToPng() succeeded');
         return resolve(data);
      }).catch(err => {
         console.log('svgToPng() returned error: ' + err)
         return reject(err.stack);
      });
   });
};

We have a set of svg's that we have tested this code against, and it works perfectly on our smaller svgs (less than 5.5MB), but it fails consistently with our larger svgs (larger than 12.5MB), with the error: Error: Input buffer has corrupt header.
We have double and triple checked our svg headers, and made sure that there weren't any problems ("<svg" tag was in first 300 characters, nothing crucial missing), and we have found that the headers of the succeeding svgs are identical to those of the failing svgs - the only difference is size.
We have tried manipulating a number of settings, such as changing the sharp cache, altering the density property, adding .resize() with various smaller sizes before the .png() command. None of these changed the behavior of the code. We even replaced all our sharp commands with:
...
await sharp(buffer)
   .metadata()
   .then(data => {
      console.log(data);
   });
...

Still, the behavior was completely unchanged. Large files failed with the corrupt header warning, and smaller files worked just fine.
Our use case is that we have users uploading large dxf or svg files to AWS S3, and we wish to quickly convert them to pngs and upload those pngs back to AWS S3.

Hello, are you able to share an example image that fails in this manner?

Hello, thank you for the fast reply!
https://s3.amazonaws.com/roast-floorplans-olive/processing/ORTLIEBSLEVEL2NoBlocks_BIG.-2.svg
This is one of the offending files.

Thanks, this file has hit the XML_PARSE_HUGE limit for the XML parser used by librsvg, namely 10,000,000 bytes. The limit is there for safety/security purposes and the ability to remove this limit is not currently exposed by libvips. This would require an upstream change to libvips to allow setting the RSVG_HANDLE_FLAG_UNLIMITED flag.
From what I can tell, the API provided by librsvg to remove this limit might mean quite a few changes to the way libvips handles SVG files.

Ah, that clears some things up! Thank you for the explanation, we just weren't making any headway trying to debug that ourselves.
We will try to do some optimizations on our svgs, and see if we can't reduce their size, and, failing that, the library svg2png appears to handle the larger files decently. It's a bit of a bummer, because we wanted to use the same library for conversion as we do for handling the resulting pngs, but not an insurmountable problem.
Thanks again!

One last question. In trying various alternatives, we considered reading the large svgs into sharp as a stream. Would this bypass the 10,000,000 byte limit?

Sadly the use of Stream-based input won't help as librsvg processes files as a whole.
If you've not already tried it, I can recommend svgcleaner to losslessly pre-process SVG files. Your original example is now responding with a 403 status so I found a similarly-sized example, which svgcleaner was able to reduce by ~75% using its default settings.
$ curl -O https://upload.wikimedia.org/wikipedia/commons/1/1b/UK_vehicles_flows_2015.svg

$ vipsheader UK_vehicles_flows_2015.svg
vipsheader: glib: Error domain 1 code 1 on line 252563 column 1 of UK_vehicles_flows_2015.svg: internal error: Huge input lookup

$ svgcleaner UK_vehicles_flows_2015.svg UK_vehicles_flows_2015.clean.svg

$ vipsheader UK_vehicles_flows_2015.clean.svg
UK_vehicles_flows_2015.clean.svg: 1884x2792 uchar, 4 bands, srgb, svgload

$ ls -l UK_vehicles_flows_2015.*
 2594226 UK_vehicles_flows_2015.clean.svg
10654533 UK_vehicles_flows_2015.svg

Thank you! We were also able to clean up and lighten our svgs, and now they work like a charm.

Hi @lovell,
With the following code and a jpeg of 88KB
exports.handler = (event, context, callback) => {
  const buffer = Buffer.from(event.body, 'base64');
  sharp(buffer)
    .jpeg(options).toBuffer()
    .then(function (data) {

I have also a quite similar error
{
    "errorMessage": "Input buffer has corrupt header",
    "errorType": "Error",
    "stackTrace": []
}

Do you have any idea ?
Regards,

@waghanza This issue relates to SVG support in sharp. Your issue looks like an unrelated AWS encoding problem. Please open a new issue with an example image and standalone (i.e. no AWS dependency) code sample that produce this behaviour if you're still having problems.

Related libvips issue at https://github.com/libvips/libvips/issues/1354

The future libvips v8.9.0 will provide an unlimited flag that sharp can elect to expose at that point - see https://github.com/libvips/libvips/issues/1354

librsvg has recently been reducing its dependence on libxml2, instead (pre)processing XML using Rust, and in newer releases it looks like this problem has pretty much gone away.
For example the following 40MB input file can now happily be processed by the current/latest version of sharp/libvips/librsvg.
https://commons.wikimedia.org/wiki/File:Location_map_San_Francisco_Bay_Area.svg
I'm going to close this as it's unlikely we'll need to expose this librsvg setting and I suspect it will be removed very soon.


此插件不兼容IE11 出错语句为  var v = t=>!(!t || !t.Window) && t instanceof t.Window;

此插件不兼容IE11 出错语句为  var v = t=>!(!t || !t.Window) && t instanceof t.Window;

English only please

var v = t=>!(!t || !t.Window) && t instanceof t.Window  This sentence reported an error in ie11 browser

The example you wrote runs on ie11 browser and also reports an error

Sorry, IE11 is not supported.

I hope you update the component to support ie11, please

Sorry，My English is so poor

var v = t=>!(!t || !t.Window) && t instanceof t.Window This sentence reported an error in ie11 browser

@gmsa
I am also getting the same error in IE11. Is there any fix for this?

Version 2.3.12-legacy supports IE

Version 2.3.12-legacy supports IE


[process exited with code 1 (0x00000001)]

Hi,
As you can tell, I am not Python programmer, any assistance with the below would be greatly appreciated.
"Traceback (most recent call last):
File "C:\Temp\GmailBruterV2-master\GmailBruterV2.py", line 175, in
StartShell()
File "C:\Temp\GmailBruterV2-master\GmailBruterV2.py", line 68, in StartShell
with open(os.path.join("data" , "Commands") ,'r') as CommandsFile:
FileNotFoundError: [Errno 2] No such file or directory: 'data\Commands'
[process exited with code 1 (0x00000001)]"
Thanks
Britts

Can you please provide the command you used to run the tool along with your os type and the current path you ran the program from so I can check if this's a global issue or it happened from something wrong you did

Can you please provide the command you used to run the tool along with your os type and the current path you ran the program from so I can check if this's a global issue or it happened from something wrong you did

Same question


zsh completion: provide container names in addition to ids

[adrian@pluto:~/dock]> docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                     NAMES
1238d09b7774        training/webapp     "python app.py"     11 minutes ago      Up 11 minutes       <IP_ADDRESS>:32768->5000/tcp   mad_fermat
[adrian@pluto:~/dock]> docker port
containers
1238d09b7774  --      11 minutes, training/webapp
webapp

Instead of the ID it would be nice if the name was in the list, too.
Also, the webapp in the second line probably shouldn't be there:
[adrian@pluto:~/dock]> docker port webapp
Error response from daemon: no such id: webapp

Hi!
Please read this important information about creating issues.
If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.
If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.
This is an automated, informational response.
Thank you.
For more information about reporting issues, see https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues

BUG REPORT INFORMATION
Use the commands below to provide key information from your environment:
docker version:
docker info:
uname -a:
Provide additional environment details (AWS, VirtualBox, physical, etc.):
List the steps to reproduce the issue:
1.
2.
3.
Describe the results you received:
Describe the results you expected:
Provide additional info you think is important:
----------END REPORT ---------
#ENEEDMOREINFO

Thanks for reporting
/cc @sdurrheimer @vincentbernat (who maintain the zsh completion scripts <3)

I think this has been fixed by a recent PR. @ThiefMaster, could you test again with the current version?

Latest version can be found here if you want to try it: https://github.com/docker/docker/blob/master/contrib/completion/zsh/_docker

Works fine :)

Thanks for testing @ThiefMaster! I'll close this issue then :+1:


Issue with "update"

If I load a form with a date already in the field and run the update method and then I open the picker and click off of the calendar, so it closes, the field value gets removed.

nevermind. User issue.


Question - Creating custom spans in functions

I have just started to explore distributed tracing and have a question regarding this.
I want to create a Span inside the parseDonutInfo() to trace the time taken to complete the function call.
https://github.com/opentracing-contrib/java-opentracing-walkthrough/blob/master/microdonuts/src/main/java/com/otsample/api/ApiContextHandler.java#L85
In order to do that, I create a span -
try (Span parseDonutsInfoSpan =
                GlobalTracer.get().buildSpan("parseDonutsInfo_span")start()) {}

However, this span doesn't get included along with the parent Span. It is present as a separate trace in Zipkin UI. How do I ensure that doPost function is the parentSpan of parseDonutsInfo().
Thanks, Let me know if you need any more details.

AFAIU since the parent span was put as an attribute on the request in doPost (just before the call to parseDonutsInfo), you could just do this:
        static DonutRequest[] parseDonutsInfo(HttpServletRequest request)
            throws IOException
        {
            Span parentSpan = (Span) request.getAttribute("span");
            Span parseDonutsInfoSpan = GlobalTracer.get().buildSpan("parseDonutsInfo_span").asChildOf(parentSpan)
                .start();
          ...
          parseDonutsInfoSpan.finish();

Thanks for the reply @jorgheymans . My question is slightly different, though I may have not worded it properly. How do I ensure that a Span is included as a part of the common GlobalTracer without explicitly passing the parentSpan.
Scenario -
If I want perform a trace = doPost() -> func1() -> func2() -> DB call -> Return. If I were to achieve this trace, should I pass the respective parentSpan across each of these operations?

According to https://github.com/opentracing/opentracing-java#accessing-the-tracer  you should be ok to use GlobalTracer.get().activeSpan() , and pass that in asChildOf().

<artifactId>opentracing-api</artifactId> is not included in the pom. @bhs it'll be helpful if you can share some insight on this. Thanks!

@kvsurii I'm sorry for the delay, just noticed this thread via a separate search!
If adding the opentracing-api artifact resolves the issue here, we should just make that change. From skimming the above, I believe the problem may instead be resolved by the activeSpan() functionality... that way you can take advantage of TLS and save the trouble of manually passing the Span instance around.

Thanks for the comment @bhs ! It looks like https://github.com/opentracing-contrib/java-opentracing-walkthrough/pull/3 is using activeSpan(). I'll also try it out from my end.


Warning after HA core 2024.2

Logger: py.warnings
Source: loader.py:864
First occurred: 08:09:49 (1 occurrences)
Last logged: 08:09:49
/config/custom_components/solarman/sensor.py:191: SyntaxWarning: invalid escape sequence '\S' if uom and (re.match("\S+", uom)):

The warning you're seeing is because Python is treating \S as an invalid escape sequence within the regular expression. In Python strings, the backslash \ is used as an escape character, so \S without a preceding r (indicating a raw string) is problematic.
To fix this, you should use a raw string for your regular expression. This way, backslashes are treated literally, not as escape characters. Here's the corrected line of code:
python
if uom and (re.match(r"\S+", uom)):
By adding the r before the string, it tells Python to treat the backslashes as literal characters. This should resolve the SyntaxWarning.


Incomplete XML generated. (Software List iq128.xml)

Very simple bug.

SabreTool ud iq128.xml

It makes incomplete XML.
software list tag not closed.

Should be supported now.


Update installation.rst

Somewhere in 2015 a requirement for a certain https://python-pillow.github.io/ version made it so that the development headers for jpeg are also needed. This was tested on ubuntu 14.04 raspberry pi 2 model B. Error log will be up for a month http://pastebin.com/R1S17NCk

Should now be installed by splash


fix: wrong variable name

Rename variable
we have < less then so it is plus one

Why ❔

Checklist

[ ] PR title corresponds to the body of PR (we generate changelog entries from PRs).
[ ] Tests for the changes have been added / updated.
[ ] Documentation comments have been added / updated.

@kelemeno main is for audited / production code. While this change does not impact the execution, it should still go to the dev branch


Python3 update branch

In this branch are three new files that should be sufficient for the update to Python 3. In the align folder, you will find "prepare_transcripts_p3.ipynb" and "calculate_alignment_p3.ipynb." Both have been extensively tested and work well with our tutorials. The changes to get them compatible were quite minor (I have a file that records all the changes if you are interested in seeing it). The other important file is "requirements_py3.txt." With this update, I made things compatible with the latest versions of nltk, gensim, pandas, etc. The exact specifications are listed.

Please let me know if anything else is needed for the PyPI update!

I think there was some confusion. There was no need to correct the deprecated issues as these were corrected in the original files with "_p3" in the name. See my original comment at the top of this page (from back in September).
Moving forward, to make things more sensible, I have removed the previous files and gave them more explicit names. Now, the files in the "align" folder that end in "public.py" are the final Python 3 files that can be added to pip and can added to Github (just remove the "public" in the filename). The files in the "align" folder that end in "private.pynb" are notebooks of the "public.py" versions that you can use for testing. However, if confusing, just remove the notebooks. These will have to be removed eventually when we merge with the main repo.
One thing to note, there is a critical difference in the private and public versions of the "prepare_transcripts" file. The public version uses Gutenberg which is specified on lines 485-492. Specifically it uses the lines:
module_path = os.path.dirname(os.path.abspath(file))
then construct the path to the text file
training_dictionary = os.path.join(module_path, 'data/gutenberg.txt')
I take it this has something to do with pip? When running the program locally (private notebooks) I changed the code here so I can run the "big.txt" dictionary.
Lastly, the public and private files are completely compatible with Python 3 and the other packages specified in the updated requirements.txt file.
Once we have these new files up on pip as v0.0.5, I can test the notebook tutorials in the "examples" folder to ensure we are all good to go with this Python 3 version.

Hi, Nick,
As I noted in my email earlier, I think some of the confusion happened because I was directly building and testing the whole package rather than using the interactive ipynb so that we could test the whole thing before shipping it. I hadn't realized that those code changes hadn't been integrated into the .py files as well.
(But if you did include them as separate p3.py files, I think they may not have made it into the commits—I haven't seen them in any of the files I've been using.)
As I tried to do last week, I've been actively testing the code with the notebooks in the examples directory. They're looking pretty good! So far, only a few tweaks around print (which now requires parentheses).
More soon!
Cheers,
Alex


⚠ Bofrost.de has degraded performance

In 7043601, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1863 ms

Resolved: Bofrost.de performance has improved in b80a8c7 after 1 hour, 14 minutes.


Use cross-spawn to support Windows

Tests run on Windows:
> npm t
> tsc && jest --no-cache

 PASS  test/test.ts (22.957 s)
  √ CloudFormation Test (4137 ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   1 passed, 1 total
Time:        23.306 s
Ran all test suites.

@micro-jumbo thank you for contribution 👍


[Operator] Add upsample_nearest2d op [MooreThreads]

PR Category

Operator upsample_nearest2d
Type of Change

New Feature
Description

https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/native_functions.yaml#L12777C6-L12777C7
func: upsample_nearest2d(Tensor self, SymInt[2] output_size, float? scales_h=None, float? scales_w=None) -> Tensor
Issue

Progress

[ ] Change is properly reviewed (1 reviewer required, 2 recommended).
[ ] Change is responded to an issue.
[ ] Change is fully covered by a UT.

Performance

PTAL @Bowen12992


Click and sync

Allow the user to synchronize the inbox by clicking on the systray icon: a simple left click will do the job.

This feature cannot be implemented because the double click on the systray icon is already doing an action. Binding the simple click will do the job twice when the user will double clicked the systray icon.


Enable opcache file to improve CLI scripts speed

As all the GAE deployments are atomic, we can compile with the flag --enable-opcache-file, and the user choose if he wants or not to enable it on the php.ini file.

I signed it!


Parallel execution error on version 1.2.1 - "Maximum call stack size exceeded"

What are you trying to achieve?
Run multiple tests in parallel
What do you get instead?
RangeError: Maximum call stack size exceeded
at normalizeStringWin32 (path.js:33:30)
at Object.resolve (path.js:338:20)
at sync (C:...\npm\node_modules\codeceptjs\node_m
odules\mkdirp\index.js:68:14)
at sync (C:...\npm\node_modules\codeceptjs\node_m
odules\mkdirp\index.js:77:24)
at sync (C:...\npm\node_modules\codeceptjs\node_m
odules\mkdirp\index.js:78:17)
at sync (C:...\npm\node_modules\codeceptjs\node_m
odules\mkdirp\index.js:78:17)
at sync (C:...\npm\node_modules\codeceptjs\node_m
odules\mkdirp\index.js:78:17)
at sync (C:...\npm\node_modules\codeceptjs\node_m
odules\mkdirp\index.js:78:17)
at sync (C:...\npm\node_modules\codeceptjs\node_m
odules\mkdirp\index.js:78:17)
at sync (C:...\npm\node_modules\codeceptjs\node_m
odules\mkdirp\index.js:78:17)
path.js:33
function normalizeStringWin32(path, allowAboveRoot) {
Seems related to #1068
Details

CodeceptJS version: 1.2.1
NodeJS Version: 8.9.4
Operating System:  Windows 7
WebDriverIO Version: 4.12.0
Configuration file:

{
  "output": "./output",
  "helpers": {
    "WebDriverIO": {
      "url": "url",
      "browser": "chrome",
      "host": "localhost",
      "port": "4444",
      "coloredLogs": true,
      "restart": false,
      "windowSize": "maximize",
      "keepCookies": true,
      "smartWait": 5000
    }
  },
  "multiple": {
    "blah": {
      "chunks": 2,
      "grep": "@blah"
    }
  },
  "include": {
    ...
  },
  "mocha": {},
  "bootstrap": false,
  "teardown": null,
  "hooks": [],
  "tests": "./tests/**/*/*_test.js",
  "timeout": 10000,
  "name": "codecept"
}

Have you tried adding ’"browsers": ["chrome"]’ to your blah config in multiple?

@sveneisenschmidt  I tried that just now and get the same error.
"multiple": {
    "blah": {
      "chunks": 2,
      "browsers": [ "chrome" ],
      "grep": "@blah"
    }
  }

This appears to be relate to issue #1084 I reported a few weeks back.
In either case the error seem to be throw in call to mkdirp.  I added some logging to my local mkdirp index.js file as follows:
// starts at line 84
            default:
                var stat;
                try {
                    console.log(p); // added this
                    stat = xfs.statSync(p);
                    console.log(stat); // added this
                }
                catch (err1) {
                    console.log(err1); // added this
                    throw err0;
                }
                if (!stat.isDirectory()) throw err0;
                break;

What I found was that this code seems to be being called repeatedly until call stack size max is reached. Because of the way that mkdirp works, you might either get the call stack error or the false error of 'directory already exists` as reported in my issue, which gets thrown in the catch block when the call stack size error is triggered.
There is open issue in mkdirp library here - https://github.com/substack/node-mkdirp/issues/129 - so not sure that this is CodeceptJs issue at this point.

I have identified root cause.  The colon in the output directory (i.e. parallel:*) is not valid Windows character, causing the mkdirp call to always throw ENOENT which causes infinite recursion and the max call stack size error that is ultimately thrown.  This is not mkdirp library error, but rather something that should be changed in this library to make sure invalid Windows path names are not being specified.  I will put in PR to fix the issue.

@mikecbrant @DavertMik Any idea when this fix will be released? Is there some sort of release schedule? Anxiously awaiting the fix for this one and #1069. :smile:  Thanks so much!

This is working for me now after upgrading to 1.3.0. Thank you

Just a quick thing because I have read that other people had the same issue: I added a chrome extension to my WDIO test by adding it base64 encoded to capabilities.chromeOptions.extensions. Due to that, I saw the following error on each exit though: Maximum call stack size exceeded, even when the feature didn't have any errors. The solution was to remove the extension again from the capabilities before spawning, like so:
    before: (capabilities) => {
        // We have to remove the base64 encoded chrome browser extensions as
        // the object WDIO carries along otherwise takes up too much memory
        delete capabilities.chromeOptions.extensions
}


Update index.rst

Fix warm start example.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thanks for catching this!
The TravisCI build failure is of course unrelated (and has since been fixed).


update cypress to v12.0.2

fixes https://github.com/graphql/graphiql/issues/2879

interesting, the only failure is just an error changing, i think for essentially the same error though?

@acao yeah, seems like it is the same error but printed differently, will take a look later 🙂

@B2o5T looks like requests are failing, is there a change that needs to be made to specify the url differently now?

I will retry this week!


Azure B2c - getAllAccounts() account.username is empty after the very first signup

Core Library
MSAL.js (@azure/msal-browser)
Core Library Version
2.38.2
Wrapper Library
MSAL React (@azure/msal-react)
Wrapper Library Version
1.5.11
Public or Confidential Client?
Public
Description
Hello there,
My application uses Azure B2C with a custom policy and msal-react wrapper.
The very first time the user signup and get redirected to App, the property account.username is always an empty string (account are retrieved using pca.getAllAccounts()).
As soon as the user logs out from the App, and log back in, username is correctly populated.
I understand username is assigned using emails claim in the token, in fact my custom policy has the claim emails specified in all the TechnicalProfiles.
The workaround I'm using is to call pca.logoutRedirect() when username property is empty so that everything works fine after the user log back in, but this doesn't sound correct.
Anyone can confirm this should not be the case?
Cheers
Error Message
N/A
Msal Logs
N/A
MSAL Configuration
xport const authConfiguration: Configuration = {
	auth: {
		clientId: process.env.REACT_APP_AD_CLIENT_ID as string,
		authority: process.env.REACT_APP_AD_AUTHORITY as string,
		redirectUri: window.location.protocol + '//' + (window.location.host || window.location.host),
		knownAuthorities: [process.env.REACT_APP_AD_BASE_AUTHORITY as string],
		authorityMetadata: process.env.REACT_APP_AD_AUTHORITYMETADATA!
	},
	cache: {
		cacheLocation: 'localStorage',
		storeAuthStateInCookie: false
	}
};

Relevant Code Snippets
export const getAccessToken = async (): Promise<IAccessToken> => {
	const accounts = pca.getAllAccounts();

	if (accounts.length > 0) {
		const request = {
			scopes: azAPITokenRequestScope,
			account: accounts[0],
		};

		const accessToken = await pca
			.acquireTokenSilent(request)
			.then((response: AuthenticationResult) => {
				return response.accessToken;
			})
			.catch(error => {
				authlog(error);
				if (error instanceof InteractionRequiredAuthError) {
					// fallback to interaction when silent call fails
					authlog('Error in getting token: %o', error);
					pca.acquireTokenRedirect(request);
				} else {
					pca.logoutRedirect();
				}
			});

		if (accounts[0].username === '') {
			authlog('Username of first Accounts is empty, will logout');
			pca.logoutRedirect();
		}

		return {
			accessToken: accessToken,
			account: accounts[0]
		};
	} else {
		authlog('Users array empty, will try to refresh');

	}

	return {};
};

Reproduction Steps

User sign up
User is redirected to my app but username property is empty
Once the user has logged out and logged back in username is populated correctly

Expected Behavior
username should be there after the sign up
Identity Provider
Azure AD / MSA
Browsers Affected (Select all that apply)
Chrome, Firefox, Edge, Safari
Regression
No response
Source
External (Customer)

This should not be the case. I would first confirm that the token returned on that first login also does not contain the emails claim and if so I would recommend raising with B2C Support to understand why this might be happening. MSAL doesn't have control over the claims returned so we unfortunately can't be of much assistance resolving this.


[extracting words from table]

Issue: Words Extracted Too Closely Together in Tables with PDFMiner
Problem Description
When extracting text from a PDF document containing tables with PDFMiner, it seems that the words inside the table are being extracted incorrectly. The words are all closely spaced together, making the extracted text unusable. This negatively affects our ability to properly process the information within these tables.
Steps to Reproduce

Call extract_text_to_fp on given pdf :
document.pdf
Observe that the extracted words are closely spaced together in xml file :

These pasted words correspond to a table on the page for which the references are indeed very close

Expected Behavior
We expect PDFMiner to correctly extract text inside tables, preserving the spacing between words, so that the extracted text is usable.
This issue significantly impacts our ability to correctly extract and use data contained within PDF document tables. Any assistance in resolving this problem would be greatly appreciated.
I thank you for the time spent studying this case and hope that a solution can be put in place :)

A few suggestions -
1 - Don't use an output type of XML; the comments in high_level.py specifically say only 'text' works properly.
2 - If you are extracting text using extract_text_to_fp as a library call, the exact call signature would be helpful to diagnose what is occurring. In particular, what LAParams are you passing to the call? Typically, you would tweak the LAParams for the nature of your document, specifically the narrow spacing in your case. The kind of parameters you can specify in an LAParams object are:
class LAParams:
    """Parameters for layout analysis

    :param line_overlap: If two characters have more overlap than this they
        are considered to be on the same line. The overlap is specified
        relative to the minimum height of both characters.
    :param char_margin: If two characters are closer together than this
        margin they are considered part of the same line. The margin is
        specified relative to the width of the character.
    :param word_margin: If two characters on the same line are further apart
        than this margin then they are considered to be two separate words, and
        an intermediate space will be added for readability. The margin is
        specified relative to the width of the character.
    :param line_margin: If two lines are are close together they are
        considered to be part of the same paragraph. The margin is
        specified relative to the height of a line.
    :param boxes_flow: Specifies how much a horizontal and vertical position
        of a text matters when determining the order of text boxes. The value
        should be within the range of -1.0 (only horizontal position
        matters) to +1.0 (only vertical position matters). You can also pass
        `None` to disable advanced layout analysis, and instead return text
        based on the position of the bottom left corner of the text box.
    :param detect_vertical: If vertical text should be considered during
        layout analysis
    :param all_texts: If layout analysis should be performed on text in
        figures.
    """

3 - If you are uncomfortable with the above, pdfminer comes with a CLI tool, pdf2txt.py, which you can call on the file whose text you wish to extract, with such parameters as --word-margin, which allows you to define when pdfminer will interpret a physical space between characters as whitespace chars.
In general, when dealing with idiosyncratic documents, one has to simply try different input parameters until the right combination is found for your particular document that allows you to extract the text in a sensible way.


[HotFix] REST server: remove expires in JWT payload of unit test

Cherry-picked from #2263

Coverage remained the same at 52.627% when pulling 1f194af05ec5bd0cb493840e21e6f8724e25d8c1 on qixcheng/rest-server/ut-token-expires into 879eec05287b4febbffe9fa1141b6c58c16c6511 on pai-0.10.y.


CI: Fix Selenium test failures

We should fix the following failures and errors reported by Selenium tests
2022-03-04T16:41:17.6812741Z [ERROR] Failures:
2022-03-04T16:41:17.6829347Z [ERROR]   CommonTests.addMultipleTagsCheck:196 expected [11] but found [0]
2022-03-04T16:41:17.6830223Z [ERROR]   CommonTests.searchNotShowingResultsCheck:409 No search results found
2022-03-04T16:41:17.6831261Z [ERROR]   CommonTests.tagFilterCountCheck:360 expected [-1] but found [0]
2022-03-04T16:41:17.6831943Z [ERROR]   DashboardDetailsPageTest.editChartDescription:180 Description not updated
2022-03-04T16:41:17.6832702Z [ERROR]   DashboardServiceTestPage.checkConnectionConfigTab:137 Error while updating service
2022-03-04T16:41:17.6834256Z [ERROR]   DatabaseServicePageTest.checkConnectionConfigTab:159 Error while updating service
2022-03-04T16:41:17.6835083Z [ERROR]   MyDataPageTest.checkRecentlyViewed:274 expected [dim_address] but found [shopify]
2022-03-04T16:41:17.6836601Z [ERROR]   PipelineDetailsPageTest.editTaskDescription:187 Description not updated
2022-03-04T16:41:17.6837297Z [ERROR]   RolesPageTest.addRole:90 Role not added
2022-03-04T16:41:17.6838647Z [ERROR]   RolesPageTest.addRuleWithoutOperation:222->addRole:90 Role not added
2022-03-04T16:41:17.6839440Z [ERROR]   TagsPageTest.addTagWithExistingName:231 expected [Not Used] but found [1]
2022-03-04T16:41:17.6840850Z [ERROR]   TeamsPageTest.addAsset:166->createTeam:102 Team not added
2022-03-04T16:41:17.6841540Z [ERROR]   TeamsPageTest.addUser:112->createTeam:102 Team not added
2022-03-04T16:41:17.6842588Z [ERROR]   TeamsPageTest.checkTeamsFilterCount:214->createTeam:102 Team not added
2022-03-04T16:41:17.6843167Z [ERROR]   TeamsPageTest.createTeam:102 Team not added
2022-03-04T16:41:17.6845121Z [ERROR]   TeamsPageTest.editDescription:148->createTeam:102 Team not added
2022-03-04T16:41:17.6845993Z [ERROR]   TeamsPageTest.ownerNameIsConsistentCheck:195->addAsset:166->createTeam:102 Team not added
2022-03-04T16:41:17.6846533Z [ERROR] Errors:
2022-03-04T16:41:17.6857825Z [ERROR]   CommonTests.overviewLinksAfterTour:317 » Timeout Expected condition failed: wa...
2022-03-04T16:41:17.6859024Z [ERROR]   CommonTests.tourStepSkippingCheck:329 » Timeout Expected condition failed: wai...
2022-03-04T16:41:17.6859940Z [ERROR]   DatabaseServicePageTest.checkIngestionTab:132 » Timeout Expected condition fai...
2022-03-04T16:41:17.6860711Z [ERROR]   MyDataPageTest.checkFollowingTab:260 » NoSuchElement no such element: Unable t...
2022-03-04T16:41:17.6861565Z [ERROR]   RolesPageTest.checkDuplicateRoleName:206 » Timeout Expected condition failed: ...
2022-03-04T16:41:17.6862392Z [ERROR]   RolesPageTest.editDescription:107 » Timeout Expected condition failed: waiting...
2022-03-04T16:41:17.6865429Z [ERROR]   TableDetailsPageTest.checkBreadCrumb:275 » NoSuchElement no such element: Unab...
2022-03-04T16:41:17.6866883Z [ERROR]   TableDetailsPageTest.checkFrequentlyJoinedColumns:320 » Timeout Expected condi...
2022-03-04T16:41:17.6867789Z [ERROR]   TableDetailsPageTest.checkFrequentlyJoinedTables:309 » Timeout Expected condit...
2022-03-04T16:41:17.6868492Z [ERROR]   TeamsPageTest.teamsWithSameDisplayNameCheck:251 » NoSuchElement no such elemen...
2022-03-04T16:41:17.6869164Z [ERROR]   TopicDetailsPageTest.checkFollow:112 » NoSuchElement no such element: Unable t...
2022-03-04T16:41:17.6869524Z [INFO]
2022-03-04T16:41:17.6874449Z [ERROR] Tests run: 161, Failures: 17, Errors: 11, Skipped: 0

@vivekratnavel these failures are expected. The fix is required from UI/backend side. The tickets are present for the same.

@kshinde2512 is this fixed? can we remove if the fix is needed on the backend/UI

no longer going to use selenium, going with cypress instead, so closing this issue.


Create Handout for Midterm

Issue description
Describe your issue in detail here
Look miro for more information

https://docs.google.com/document/d/1Z9hRvfIOCbnf8O5McZOE2edDp_9aXuEqhZqcdYzE0Ac/edit#
Definition Of Ready (DoR)
This issue can be worked on if

[x] a developer is assigned
[x] the issue is labeled
[x] the issue is assigned to the current sprint
[x] the issue is estimated
[x] a meaningful title has been set
[x] the issue is described in detail

We need to add the working hours

If Handout is done, please provie the PDF file in the right discord channel, and we maybe should also save the PDF in our Wikirepo as a file.


Remove code-generator scripts and generated client

What this PR does / why we need it:
Clean-up unused code.
Which issue(s) this PR fixes (optional, in fixes #<issue number> format, will close the issue(s) when PR gets merged):
Fixes #
Special notes for your reviewer:
Optional Release Note:

NONE

After running go mod tidy in this branch there are uncommitted changes
$ git st
## clean-up-generated-client
 M go.mod
 M go.sum

After running go mod tidy in this branch there are uncommitted changes
$ git st
## clean-up-generated-client
 M go.mod
 M go.sum

good catch, updated.


Multiple registry #3932

What is the purpose of the change
support ##3932
Brief changelog

Add multiple registry module and MultipleRegistry
Add Dynamic Refresh Abstract Class

Verifying this change
XXXXX
Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a GITHUB_issue field for the change (usually before you start working on it). Trivial changes like typos do not require a GITHUB issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[ ] Format the pull request title like [Dubbo-XXX] Fix UnknownException when host config not exist #XXX. Each commit in the pull request should have a meaningful subject line and body.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add sample in dubbo samples project.
[ ] Run mvn clean install -DskipTests=false & mvn clean test-compile failsafe:integration-test to make sure unit-test and integration-test pass.
[ ] If this contribution is large, please follow the Software Donation Guide.

https://github.com/apache/incubator-dubbo/issues/3599
https://github.com/apache/incubator-dubbo/issues/3084

Codecov Report

Merging #4066 into master will increase coverage by 0.16%.
The diff coverage is 63.88%.

@@             Coverage Diff              @@
##             master    #4066      +/-   ##
============================================
+ Coverage     62.72%   62.88%   +0.16%
- Complexity      474      506      +32
============================================
  Files           766      768       +2
  Lines         32860    33043     +183
  Branches       5164     5219      +55
============================================
+ Hits          20611    20780     +169
- Misses         9859     9862       +3
- Partials       2390     2401      +11

Impacted Files
Coverage Δ
Complexity Δ

...bbo/registry/multiple/MultipleRegistryFactory.java
100% <100%> (ø)
2 <2> (?)

...ache/dubbo/registry/multiple/MultipleRegistry.java
63.38% <63.38%> (ø)
21 <21> (?)

...dubbo/common/status/support/LoadStatusChecker.java
46.15% <0%> (-3.85%)
0% <0%> (ø)

...ting/exchange/support/header/HeartbeatHandler.java
83.72% <0%> (-2%)
0% <0%> (ø)

.../remoting/transport/netty4/NettyClientHandler.java
68.42% <0%> (-1.23%)
0% <0%> (ø)

...he/dubbo/remoting/transport/netty/NettyClient.java
66.1% <0%> (-1.14%)
11% <0%> (ø)

...e/dubbo/remoting/transport/netty4/NettyClient.java
70.58% <0%> (-1.06%)
0% <0%> (ø)

...ache/dubbo/remoting/transport/mina/MinaClient.java
58.46% <0%> (-0.92%)
9% <0%> (ø)

.../main/java/org/apache/dubbo/qos/textui/TTable.java
83.79% <0%> (-0.87%)
0% <0%> (ø)

...n/java/org/apache/dubbo/common/utils/UrlUtils.java
77.81% <0%> (-0.2%)
0% <0%> (ø)

... and 22 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ccb990f...df4f4e2. Read the comment docs.


Must understand issues

In this brazen new world of technological wonder (not really), you must understand this fascinating new issues tab! explore the features and reinforce your idea of the "issues" tab!

what are the features to this glorious new tab?

Interesting, it seems that by adding a label an event was added!

We can link projects as well, we can try it in another issue


Read the x,y,z coordinate values from Gcode

Hello, I would like to ask how can I read the x,y,z coordinate values from Gcode and save it as txt format?


feature/OP-19675-Gate-Master adding new parameters for argo environme…

https://devopsmx.atlassian.net/browse/OP-19675
adding parameters for argo environments api

This PR  triggered this build in this execution, please wait for it  and child pipelines to succeed before merging:  https://isd.prod.opsmx.net/deck/#/applications/dev-oesaona-4x/executions/details/01GVN21AYGSDQWS6SHTDK6RW57


LavaBrig Dungen needs a full rework

Description

This can easily hand out security encription keys, clothing, armor, it's a part of the issue of why salvage is getting on level with security on gear, at least on the armor and headset side, also clothing.
Instead it should be a full prison with orange jumpsuits and no sec gear.
Reproduction

Screenshots

Additional context

It even has a secfab


Getter for DMChannel in User.dart returns null

Future<DMChannel> get dmChannel in User.dart @row 79 returns null after findeOne call.
Basically I'm trying to get the dm channel of a user from context;
var dmChan = await ctx.member.dmChannel;
The getter seems to query channels if an Error occurs but since null the call isn't really throwing any error and a null is going to be returned.
Running from the rewrite_modular branch

Works fine now with new commit: Remove unnecessary try catch


Fixing bug #34: Idle detection not consistent in IE11

Fixes bug #34 by specifically excluding Internet Explorer
Please check if the PR fulfills these requirements

[X] The commit message follows our guidelines: https://github.com/HackedByChinese/ng2-idle/blob/master/CONTRIBUTING.md#pr
[ ] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)

What kind of change does this PR introduce? (check one with "x")
[X] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] CI related changes
[ ] Other... Please describe:

What is the current behavior? (You can also link to an open issue here)
https://github.com/HackedByChinese/ng2-idle/issues/34
What is the new behavior?
Works in IE11
Does this PR introduce a breaking change? (check one with "x")
[ ] Yes
[X] No

If this PR contains a breaking change, please describe the impact and migration path for existing applications: ...
Other information:

Coverage remained the same at 99.36% when pulling b4d841077ec81a98adae3a48e0d9c180d534043f on wenz:patch-1 into a5e24eb58cc66a7dc251e602d7cbd7616be0c1aa on HackedByChinese:master.

@HackedByChinese could you pls merge this one?

I'm nervous the default behavior for filterEvent is a kludgy mess when it comes to dealing with the user moving the mouse, and I fear user agent detection can make it worse. This needs to be greatly simplified because it's caused other problems in the past.  However, I've kind of lost track of all the original bugs that lead to this code :(

Yeah but take a Look into his branch its just a line of Code. I tried it in my current project and it worked like a charm. Would be very happy to have this merge. But could you elaborate further why this would be a bad idea? Maybe i didnt had these edge cases.

@all having this problem while the Pull Request isn't merged consider using the pointermove event which is supported in IE11 and triggers the reset. Therefore you need to create your own DocumentIntruptsources Object and register all events as seen below. This should anyway be the way to go instead of using DEFAULT interuptsources (imo). Code could look like this
const interuptsources = new DocumentInterruptSource("click keydown mousemove pointermove");
this.idle.setInterrupts([interuptsources]);

adding pointermove doesn't work for me. Having issues with mouse move in Chrome: the timer of the timeout warning stops, but the warning stays. When this expected to be checked in?

Tested today with IE11 in a VM; it detects the click and keydown to stop the idle warning.  But it does not detect mouse movement, I still get the idle warning if I only move the mouse and do not click or press keys.


Document how to skip checks in pre-commit (fixes #156)

Fixes https://github.com/nbQA-dev/nbQA/issues/156

Codecov Report

Merging #380 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##            master      #380   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           10        10
  Lines          477       477
=========================================
  Hits           477       477

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 17b837e...fbea18a. Read the comment docs.

Nice one @caioariede , thanks!
If you have capacity, do you feel like making a little pre-commit tutorial as well? In the docs, there is this tutorial which shows how to use nbqa as a command-line tool. IMO something similar showing how to use it as a pre-commit hook would be useful, and it seems you're already well-versed in pre-commit

@MarcoGorelli I think that's possible. Mind if we do it in a separate/follow-up PR?

Yes, of course!


Fix memory leak, always clean up routes after transition

Bug arose from the changes here https://github.com/tableflip/react-native-navbar/compare/d932946a106eff91a7e63b23a9de30e7a12fd5ea...16820eaa469d5d85f08613c002aee162121271d2
Basically if there's no items in the queue then the route that has just transitioned out of view doesn't get removed from the route stack.

:eyeglasses:

Works good, things are totally unmounting all over the place. :+1:


Bug Fix

Fixed issue with downloading All Albums. Download URLs were not being formed correctly.

Long time coming, but thanks for the bug fix =)


readme: Add missing path component

Great project! Thank you for making it.
I needed to add the src path component in order have this path be valid.

I signed it!

Thanks!


Add travis badge status

Questions
Answers

Description?
Add travis badge status

Type?
improvement

BC breaks?
no

Deprecations?
no

Fixed ticket?

How to test?

Thank you for review


AM-1411: Fix sonar vulnerabilities in ORM

JIRA link (if applicable)
https://tools.hmcts.net/jira/browse/AM-1411
Change description
All the below sonar severities brought down to zero in ORM.
Blocker
Critical
Major
Minor
Info - Not fixed for better code readability
Does this PR introduce a breaking change? (check one with "x")
[ ] Yes
[X] No

Closing this PR


I can't solve url

Good day,
I can't solve url for my model on Rails 3.2.
Model:
class Post < ActiveRecord::Base
attr_accessible :content, :name, :title ,:media
if Rails.env.development?
has_attached_file :media,
:styles=> {original: '800x600'},
:storage => :google_drive,
:google_drive_credentials => "#{Rails.root}/config/google_drive.yml",
:google_drive_options => {
:path => proc { |style| "#{style}#{id}#{media.original_filename}" },
:public_folder_id=> "0Bx-14dUHOQScVjlmSExDd0NueUE"
}
end
end
Next i try solve url:
Post.last.media.url(:original)
Post Load (0.4ms)  SELECT "posts".* FROM "posts" ORDER BY "posts"."id" DESC LIMIT 1
=> "No picture"
What i doing wrong? because files upload correctly and public folder have permission for read and write for every body.
Thank you!

@marcosmartingm it looks like the uploaded file has a pretty long file name - this may be directly relted to #10

rake google_drive:authorize
rake aborted!
LoadError: cannot load such file -- google/api_client
/home/subha/.rvm/gems/ruby-2.4.2/gems/activesupport-5.1.5/lib/active_support/dependencies.rb:292:in `require'


[Python] data corruption when using group_by and aggregate on large data sets

Describe the bug, including details regarding any error messages, version, and platform.
We recently found some data corruption issues when using group_by and aggregate with large data sets.
I was able to create a minimal reproducible example:
import pyarrow as pa
import pyarrow.compute as pc

COLUMN_COUNT = 5  # <- 4 works fine, 5 causes data corruption
LENGTH = 100_000_000
data = {}
# 'index' = [0, 1, 2, ... , LENGTH-2, LENGTH-1]
data['index'] = pc.indices_nonzero(pc.if_else(True, True, pa.nulls(LENGTH, pa.bool_())))
for i in range(COLUMN_COUNT):
    # fill 'i' with i (ex: '3' = [3, 3, 3, ... 3, 3])
    data[f'{i}'] = pa.nulls(LENGTH, pa.uint64()).fill_null(i)
t = pa.table(data)  # <- create table from data
print('-------------------- ORIGINAL --------------------')
print(t)
a = t.group_by(t.column_names).aggregate([])  # <- should behave like a no-op
a = a.combine_chunks()  # <- not necessary, just improves the print formatting
print('-------------- GROUP_BY / AGGREGATE --------------')
print(a)

In this example the group_by and aggregate are set up to behave like a no-op and everything works fine with COLUMN_COUNT <= 4:
-------------------- ORIGINAL --------------------
pyarrow.Table
index: uint64
0: uint64
1: uint64
2: uint64
3: uint64
----
index: [[0,1,2,3,4,...,99999995,99999996,99999997,99999998,99999999]]
0: [[0,0,0,0,0,...,0,0,0,0,0]]
1: [[1,1,1,1,1,...,1,1,1,1,1]]
2: [[2,2,2,2,2,...,2,2,2,2,2]]
3: [[3,3,3,3,3,...,3,3,3,3,3]]
-------------- GROUP_BY / AGGREGATE --------------
pyarrow.Table
index: uint64
0: uint64
1: uint64
2: uint64
3: uint64
----
index: [[0,1,2,3,4,...,99999995,99999996,99999997,99999998,99999999]]
0: [[0,0,0,0,0,...,0,0,0,0,0]]
1: [[1,1,1,1,1,...,1,1,1,1,1]]
2: [[2,2,2,2,2,...,2,2,2,2,2]]
3: [[3,3,3,3,3,...,3,3,3,3,3]]

...but results in data corruption if COLUMN_COUNT >= 5:
-------------------- ORIGINAL --------------------
pyarrow.Table
index: uint64
0: uint64
1: uint64
2: uint64
3: uint64
4: uint64
----
index: [[0,1,2,3,4,...,99999995,99999996,99999997,99999998,99999999]]
0: [[0,0,0,0,0,...,0,0,0,0,0]]
1: [[1,1,1,1,1,...,1,1,1,1,1]]
2: [[2,2,2,2,2,...,2,2,2,2,2]]
3: [[3,3,3,3,3,...,3,3,3,3,3]]
4: [[4,4,4,4,4,...,4,4,4,4,4]]
-------------- GROUP_BY / AGGREGATE --------------
pyarrow.Table
index: uint64
0: uint64
1: uint64
2: uint64
3: uint64
4: uint64
----
index: [[0,1,2,3,4,...,3,3,3,3,3]]
0: [[0,0,0,0,0,...,4,4,4,4,4]]
1: [[1,1,1,1,1,...,10521510,10521511,10521512,10521513,10521514]]
2: [[2,2,2,2,2,...,0,0,0,0,0]]
3: [[3,3,3,3,3,...,1,1,1,1,1]]
4: [[4,4,4,4,4,...,2,2,2,2,2]]

Component(s)
Python
Version
pyarrow==12.0.1.  I downgraded to pyarrow==11, pyarrow==10, pyarrow==9, and pyarrow==8 and observed the above behavior in those versions as well.
Platform
Linux (Ubuntu 22.04 LTS)

Hmm...given how stable this is I wonder if we're overflowing an int32 somewhere.

Indeed, this was the case.  In the internals of the "group by" code is a hash map.  This hash map stores keys in row-major order using the row-format (row-major order is commonly used with hash-maps since it's expected they will be accessed in random non-sequentially).  This structure is called the RowTableImpl: https://github.com/apache/arrow/blob/9d92ed4dc4d6d691668609641043561137e91401/cpp/src/arrow/compute/row/row_internal.h#L162
In the example we are using 6 columns (if we include the index) of int8 as the hash keys.  This means each row in our row table is 48 bytes large.  The test uses 100,000,000 rows. So we end up with 4,800,000,000 bytes which is over the int32 limit of 4,294,967,296.  With 5 columns we only have 4,000,000,000 bytes and so it fits.
This should be ok.  The RowTableImpl appears to have been designed to store 2^64 bytes of data.  In practice we do not have any unit tests covering this case and so, unsurprisingly, we have mistakes.  The following patch fixes the reproduction from the OP: https://github.com/apache/arrow/commit/c901d1eacd614cdb0f44cd736034be098b16aae0
Unfortunately, I still have pretty low confidence in the robustness of this code when working with tables this large.  There are different paths for boolean fields, dictionary fields, string-like fields, and all of these paths are futher parameterized by AVX2 vs. not-AVX2 and we no longer have CI for AVX2.
We are missing a set of unit tests to cover these scenarios and I don't think I'll have time to add them for a while.  So I'm uncertain at the moment whether it would be better to put in this short-term fix or to simply reject and fail a plan once the keys table exceeds 4GB.  If anyone is interested in taking on the effort of adding these unit tests I'd be happy to explain further.  In the meantime I'm not sure when I'll get to it.  I'm marking this blocker to force myself to make some kind of decision on this in the 13.0.0 timeframe.

@westonpace do you think we should prioritize this to have a fix for 13.0.0? or is this not a blocker for the release?

@jorisvandenbossche Should this be a release blocker?

It would be good to at least change this to return an error in time for 13.0.0.  It's not a new issue but it is "invalid data" which is pretty bad.  I agree it is probably not a blocker.

Might someone with more acero knowledge have time and be willing to work on this issue? cc @ildipo @rtpsw

This issue may have already been fixed by #43389.
I'm however unfortunately having troubles running pyarrow in my local at the moment so I can't verify. If anyone can help on that (running the python test on master branch), it'll be much appreciated.

Hey @zanmato1984, I think you're right that this is now fixed on main. I can reproduce the issue with PyArrow 17 but with PyArrow built from main I no longer get the data corruption.
>>> pyarrow.__version__
'18.0.0.dev372+gfc2e018ef1.d20240925'

And when I run the script with COLUMN_COUNT = 5:
❯ python repro.py
-------------------- ORIGINAL --------------------
pyarrow.Table
index: uint64
0: uint64
1: uint64
2: uint64
3: uint64
4: uint64
----
index: [[0,1,2,3,4,...,99999995,99999996,99999997,99999998,99999999]]
0: [[0,0,0,0,0,...,0,0,0,0,0]]
1: [[1,1,1,1,1,...,1,1,1,1,1]]
2: [[2,2,2,2,2,...,2,2,2,2,2]]
3: [[3,3,3,3,3,...,3,3,3,3,3]]
4: [[4,4,4,4,4,...,4,4,4,4,4]]
-------------- GROUP_BY / AGGREGATE --------------
pyarrow.Table
index: uint64
0: uint64
1: uint64
2: uint64
3: uint64
4: uint64
----
index: [[0,1,2,3,4,...,98566139,98566140,98566141,98566142,98566143]]
0: [[0,0,0,0,0,...,0,0,0,0,0]]
1: [[1,1,1,1,1,...,1,1,1,1,1]]
2: [[2,2,2,2,2,...,2,2,2,2,2]]
3: [[3,3,3,3,3,...,3,3,3,3,3]]
4: [[4,4,4,4,4,...,4,4,4,4,4]]

Thanks a lot @amoeba for verifying. Really appreciate it!
Let's close this as fixed by #43389 :)


Need ability to represent concepts "Bearing" and "Velocity"

Background
The current UCO framework provides a robust platform for representing various cyber and physical objects but lacks specific properties for detailed geospatial analysis. Specifically, there is no representation for "Bearing" (direction of travel in degrees from 0 - 360) and "Velocity" (the speed of travel in kilometers or miles). These concepts are critical in investigations involving movement tracking, such as transportation incidents, or analyzing the movement patterns of entities.
Example
Background: An investigator is analyzing a traffic incident involving a vehicle suspected of speeding and making an abrupt turn before the collision. The vehicle is equipped with a modern infotainment system that records geospatial data, including the direction of travel (bearing) and speed (velocity).
Data Extraction: The infotainment system's data is extracted using forensic tools compatible with the vehicle's system. The data includes a timestamped log of the vehicle's bearing and velocity.
Presentation of Data:
Extracted Data Log:
Timestamp: 2023-11-22T15:30:00Z
Bearing: 90 degrees (indicating eastward direction)
Velocity: 50 km/h
Timestamp: 2023-11-22T15:32:00Z
Bearing: 135 degrees (indicating a southeast direction)
Velocity: 70 km/h
Timestamp: 2023-11-22T15:33:30Z
Bearing: 180 degrees (indicating southward direction)
Velocity: 30 km/h
Analysis:
The data shows that at 15:30, the vehicle was traveling eastward at a moderate speed of 50 km/h.
By 15:32, the vehicle's speed had increased to 70 km/h, and the bearing changed to 135 degrees, indicating a significant turn towards the southeast.
At 15:33:30, a notable change in bearing to 180 degrees and a reduction in speed to 30 km/h suggest an abrupt southward turn and deceleration, possibly correlating with the moment of the incident.
Investigative Actions:
Map Overlay: The investigator overlays this data onto a map of the incident area to visualize the vehicle's path and correlate it with physical evidence and witness statements.
Speed Analysis: The speed data is compared against the posted speed limits to ascertain if the vehicle was speeding.
Maneuver Correlation: The bearing changes are examined in relation to the road layout and traffic conditions to assess the driver's actions leading up to the incident.
Conclusion:
Based on the bearing and velocity data extracted from the infotainment system, the investigator gains insights into the vehicle's movement patterns, contributing significantly to reconstructing the events leading up to the traffic incident.
Requirements
I have two requirements.
Requirement 1
Description: Introduce a new property, uco-location:Bearing, to represent the direction of travel in degrees (0 - 360).
Type: owl:DatatypeProperty
Range: xsd:decimal
Requirement 2
Description: Introduce a new property, uco-location:Velocity, to represent the speed of travel in kilometers or miles per hour.
Type: owl:DatatypeProperty
Range: xsd:decimal
Risk / Benefit analysis
Benefits
Enables precise tracking and representation of directional movement and speed, enhancing the ontology's utility in geospatial and transportation-related investigations.
Aligns UCO with standard practices in geospatial data representation, improving interoperability with other systems and datasets.
Risks
Potential complexity in ensuring consistent unit usage (e.g., kilometers vs. miles for Velocity) across different datasets.
Additional validation mechanisms may be needed to ensure the accuracy and consistency of Bearing and Velocity data.
These are simple geo concepts that will lead to UCO eventually needing to accomodate GeoJSON concepts.
Competencies demonstrated
Enabled or Upgraded Competency with "Bearing" and "Velocity" in UCO
Competency 1
Enabled or Upgraded Competency with "Bearing" and "Velocity" in UCO
Scenario: An investigator is analyzing a traffic incident involving a vehicle suspected of speeding and making an abrupt turn before the collision. The vehicle, equipped with a modern infotainment system, has recorded crucial geospatial data, including the vehicle's bearing and velocity.
Competency Question 1.1
Question: How can we determine the vehicle's direction and speed changes at specific times to understand its movement pattern before the collision?
Result 1.1
Result: With the integration of "Bearing" and "Velocity" into UCO, investigators can now analyze the directional changes and speed variations of the vehicle. This data provides insights into the vehicle's maneuvering and speed at critical moments leading up to the incident.
PREFIX uco-location: <https://ontology.unifiedcyberontology.org/uco/location/>
PREFIX uco-time: <https://ontology.unifiedcyberontology.org/uco/time/>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>

SELECT ?timestamp ?bearing ?velocity
WHERE {
  ?vehicleEvent uco-location:bearing ?bearing ;
                uco-location:velocity ?velocity ;
                uco-time:timestamp ?timestamp .

  FILTER(?timestamp >= "2023-11-22T15:30:00Z"^^xsd:dateTime &&
        ?timestamp <= "2023-11-22T15:33:30Z"^^xsd:dateTime)
}
ORDER BY ?timestamp

Explanation
The query retrieves data for a specific vehicle's movements (?vehicleEvent) within the timeframe of the incident.
It focuses on extracting the bearing (?bearing) and velocity (?velocity) at each recorded timestamp (?timestamp).
The filtered time range corresponds to the period just before and during the incident, allowing for a focused analysis of the vehicle's behavior.
This query enables the investigator to construct a timeline of the vehicle's directional changes and speed, which is crucial for understanding the events leading up to the traffic incident.
Solution suggestion
Define Bearing Property
uco-location:Bearing a owl:DatatypeProperty ;
    rdfs:label "Bearing"@en ;
    rdfs:comment "Represents the direction of travel in degrees from 0 to 360."@en ;
    rdfs:domain uco-location:Location ;
    rdfs:range xsd:decimal ;
    .

Define Velocity Property
uco-location:Velocity a owl:DatatypeProperty ;
    rdfs:label "Velocity"@en ;
    rdfs:comment "Represents the speed of travel in kilometers or miles per hour."@en ;
    rdfs:domain uco-location:Location ;
    rdfs:range xsd:decimal ;
    .

(2023-11-27: OC Chair edited lightly to fix Markdown syntax.)
Coordination

Tracking in Jira ticket OC-300

[ ] Administrative review completed, proposal announced to Ontology Committees (OCs) on TODO-date
[ ] Requirements to be discussed in OC meeting, date TBD

[ ] Requirements Review vote has not occurred

[ ] Requirements development phase completed.
[ ] Solution announced to OCs on TODO-date
[ ] Solutions Approval to be discussed in OC meeting, date TBD
[ ] Solutions Approval vote has not occurred

[ ] Solutions development phase completed.
[ ] Backwards-compatible implementation merged into develop for the next release
[ ] develop state with backwards-compatible implementation merged into develop-2.0.0
[ ] Backwards-incompatible implementation merged into develop-2.0.0 (or N/A)
[ ] Milestone linked
[ ] Documentation logged in pending release page
[ ] Prerelease publication: CASE develop branch updated to track UCO's updated develop branch
[ ] Prerelease publication: CASE develop-2.0.0 branch updated to track UCO's updated develop-2.0.0 branch

Thank you @vulnmaster for the proposal, it's pretty well written.
I have a few initial remarks:

The "degrees" property needs to include in its definition what "0" means.  Does it mean East (Cartesian), Magnetic North, or True North?
Because this is introducing units of measure, I encourage UCO to not invent concepts here, but instead adopt any other existing implementation.  I am aware of QUDT (GitHub repo here) providing unit:KiloM-PER-HR in the unit vocabulary, as well as unit:DEG.
Because this is also covering measurements, I believe this proposal would also depend on qualities.
uco-time:timestamp is mentioned in the competency illustration, but does not exist as a property.  (I also suggest avoiding focus on the current members of the UCO time namespace, as they are stubs with currently no functional impact.)  uco-core:endTime or startTime, or uco-action:endTime and startTime should be used instead.

Which property should be used, and the significance of the gap between start and end times of measurements, is one of the reasons I've mentioned sosa:Observations complementing UCO's observable:Observations.

I saw mention of GeoJSON.  That is one of the representations that had come up as UCO was discussing alignments of the Location namespace.  GeoJSON, and any other suggested representations, should be evaluated with this requirements wiki page.

What are the issues with using the SSN Ontology?  Ultimately, won’t an Observation be based on a sensor?
I’m always a fan of subsuming well vetted ontologies where it does not constrain one’s objectives.
At least it makes “sense” to me. (sorry 😉)
Patrick Maroney | Principal – Cybersecurity | AT&T Services, Inc.
Threat Analytics
[Image]
[A blue circle with text  Description automatically generated]
From: Alex Nelson @.>
Date: Monday, November 27, 2023 at 2:24 PM
To: ucoProject/UCO @.>
Cc: Subscribed @.***>
Subject: Re: [ucoProject/UCO] Need ability to represent concepts "Bearing" and "Velocity" (Issue #567)
Re: Because this is also covering measurements, I believe this proposal would also depend on qualities. Instead of owl: DatatypePropertys, I believe bearing and velocity would be better suited as qualities. Here are the main reasons: Given this
Re:

Because this is also covering measurements, I believe this proposal would also depend on qualitieshttps://urldefense.com/v3/__https:/github.com/ucoProject/UCO/issues/535__;!!BhdT!khuTqGuEF96hST33433pTzG05v2dWV8yQ3ckkC-69t_45syDF6ZmudXVkAcTGjn_rolf8AaJUXQKYR5qzH53ityDS48$.

Instead of owl:DatatypePropertys, I believe bearing and velocity would be better suited as qualities. Here are the main reasons:
Given this SPARQL:
?vehicleEvent uco-location:bearing ?bearing ;
            uco-location:velocity ?velocity .

The query shape tells me it is not the vehicle that has a bearing and velocity, but instead the event pertaining to the vehicle that has a bearing and velocity. That seems like a confusion of a record with a spatial object, and if we continued down that road we could probably end up measuring databases in transaction-miles-per-hour.
Instead, I think your suggested uco-location:Velocity would be a owl:Class, subclass of Quality, as an object inhering in spatial things. (For instance, a uco-observable:Device always has a velocity, even non-0 if given wheels or wings.) And likewise for uco-location:Bearing, it should also be a quality.
The SOSA/SSN spinning cupshttps://urldefense.com/v3/__https:/www.w3.org/TR/vocab-ssn/*spinning-cups__;Iw!!BhdT!khuTqGuEF96hST33433pTzG05v2dWV8yQ3ckkC-69t_45syDF6ZmudXVkAcTGjn_rolf8AaJUXQKYR5qzH53I3HGL9w$ example includes a speed measurement of an object clocking in at 47 km/h.
—
Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/ucoProject/UCO/issues/567*issuecomment-1828471559__;Iw!!BhdT!khuTqGuEF96hST33433pTzG05v2dWV8yQ3ckkC-69t_45syDF6ZmudXVkAcTGjn_rolf8AaJUXQKYR5qzH532NTqcNA$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AAYSFYNUDYXW5G2JKUPFM4DYGTSFBAVCNFSM6AAAAAA7W75ZJWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRYGQ3TCNJVHE__;!!BhdT!khuTqGuEF96hST33433pTzG05v2dWV8yQ3ckkC-69t_45syDF6ZmudXVkAcTGjn_rolf8AaJUXQKYR5qzH53BcgHSC0$.
You are receiving this because you are subscribed to this thread.Message ID: @.***>

What are the issues with using the SSN Ontology? Ultimately, won’t an Observation be based on a sensor? I’m always a fan of subsuming well vetted ontologies where it does not constrain one’s objectives.

There are no notes for or against SSN+SOSA yet in UCO's discussions.  They haven't been proposed for adoption by owl:imports.
I've exercised them a bit in CASE-Corpora (SHACL shapes in this directory), so I have some sense of what they'd be like to work with.
I think UCO would need to handle Qualities before discussing SSN+SOSA.
And I am still trying to give Qualities their fair shake, but I'm working through some review mechanisms to support that.

Re:

The "degrees" property needs to include in its definition what "0" means.  Does it mean East (Cartesian), Magnetic North, or True North?

@vulnmaster , I apologize, I missed that your initial post indicates a 0-bearing is not East.  But, how should we handle Magnetic North vs. True North?


Error in https://docs.thatopen.com/Tutorials/FragmentIfcLoader

"Load IFC" and "Import Fragments" actions not working also shows an error on chrome console .

Fixed!


Update URLs to obtain data from FAOSTAT

The current URLS are not working.

The URL to obtain the Resource Land indicators are not working.

http://fenixservices.fao.org/faostat/static/bulkdownloads/Inputs_Land_E_All_Data_(Normalized).zip
This will be the URL to obtain the data.
A migration process should be done from "FAOSTAT_Importer" to "FAOSTAT-Norm_Importer"

fixed mainly in commits 7debda967a495e3bfe955c73624003012888be24 and 04b2145d0fac19c464d2ac758b2b4ee4234647f1


Adding journey details fields to AJO entity schema

@chetanmeh @akshitjain I've added 3 fields to this schema related to Journey. Please review.

@mjasrotia Please merge this in the next cycle. Thanks.


Running with a custom dataset, changes size in formatting

I have a custom dataset that I have been formatting with convert.py from https://github.com/autonomousvision/gaussian-opacity-fields so it will run with gaussian-opacity-fields, gaussian-splatting, mip-splatting. However this changes the size of the images in the "images" folder to be smaller than what they originally were. Do you believe it is best to run with the original size of the images or the smaller sizes? Either will work.

Hi, what are the sizes? For 3DGS I would recommend keeping the sizes at around 1000px. (w*h) close to 1M pixels. In nerfbaselines you only need to downscale the images, camera matrices will be rescaled automatically.

Hi @jkulhanek, more details about the issue mentioned by @Mikey-E can be found here
Please review. Thanks

From what I understand, the question is more about what convert.py in https://github.com/autonomousvision/gaussian-opacity-fields repo does? In that case the https://github.com/autonomousvision/gaussian-opacity-fields repo is a better place to ask the question.

We were just concerned if having smaller sized images (after we use convert.py to format our own dataset) will be a problem for any of the nerfbaselines models, theoretically or otherwise, from a research perspective. If you are unsure that is understandable and okay, we just thought we'd reach out for any extra insight we can get.

From the research perspective, smaller images are definitely fine.

Feel free to downscale


How to generate websocket token when using AccessToken with API?

Use Case (delete as appropriate): Using Portainer in a Commercial setup.
Have you reviewed our technical documentation and knowledge base? Yes
We have built our own web front-end pages, but the back-end uses portainer to access Docker. We have our own permission system, so we uniformly use the admin's AccessToken to access portainer's API. However, currently the portainer websocket API requires a token parameter and the browser needs to know this parameter. Obviously, we cannot leak the AccessToken to the browser, and generating a jwt token using an admin account is not secure.
Is there any solution to generate a disposable token for the browser to establish a websocket connection with the portainer server, instead of directly giving the client an administrator's jwt token?

Hello!
 You API then serves as a proxy to Portainer's API, using an Admin Access Token.
In order to use the websocket API (assuming our route /api/websocket/attach) you can create a websocket connection between your UIs and your server, using your own credentials/permission system.
Then from your server, you open a WS connection to our server ( /api/websocket/attach)
On messages from either sides, you transfer the content of the messages from one websocket to the other.
Note that even if the token is marked as mandatory in the documentation, you can skip it if you include the appropriate headers in the initiation request (either Authorization: Bearer $JWT or x-api-auth: $ACCESS_TOKEN)

OK, I got it. Thanks a lot~


The API test cant work because to get that data from data ingestion you HAVE to supply an interval

I feel like we either need to always require interval, or dont let API requests/non interval requests start data ingestion

So why does data ingestion need an interval? Is it because of the way NOAATANDC urls are setup? Thoughts below:

Yes, it seems that NOAA TAC really makes this complicated since most other websites don't usually have these interval choices.
Maybe we can apply a default interval choice for an ingestion class if they have multiple interval choices and the API can call for that default interval if they don't supply one. So maybe hourly for example.
We can leave interval as optional in the API but provide documentation with the possible interval options depending on the input source.
I know that making interval a required variable makes sense for NOAA TAC, but I don't think it will make sense for other ingestion classes which is why I think we should just apply a default case for TAC and any other future ingestion classes that might have multiple intervals.
Anyways if an end-user wants say 6-min data from TAC using our API then they will know that and we just have to provide the proper documentation on how to retrieve it.

Data ingestion classes should have a default interval if an interval is not provided

Refactor ingest_series in NOAATANDC.py to handle if seriesDescription.interval is None


NIFI-9626: Allowing Stateless NiFi to parse flow snapshots with unrec…

…ognized fields
Description of PR
This allows an older version of Stateless NiFi to run with a newer version of a VersionedFlowSnapshot.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[ ] Has your PR been rebased against the latest commit within the target branch (typically main)?

[ ] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

[ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[ ] Have you written or updated unit tests to verify your changes?
[ ] Have you verified that the full build is successful on JDK 8?
[ ] Have you verified that the full build is successful on JDK 11?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.

Thanks @gresockj that's a good change to have. Thanks. +1 merged to main.


[BUG]

Acknowledgement of preliminary instructions

[X] I have read the preliminary instructions, and I am certain that my problem has not already been addressed.

What problem did you encounter?
no matter how many times i open it the menu will not pop up it only opens roblox

Make a shortcut with the “—menu” argument

Did you read https://github.com/pizzaboxer/bloxstrap/wiki/Opening-the-Bloxstrap-menu? You were told about this before you installed Bloxstrap too.