Different values between validation/testing when sets are the same

🐛 Bug
Hi. I've come across a weird bug (?). To test out my workflow, I trained, validated and tested a video classifier on the same set. First bug, I get different accuracy and loss values between my validation and test sets (even though they are the same). Second bug, when loading the model from a check point, the test accuracy is again different. Third issue (maybe not a bug), I notice training/evaluation is slower when running from a checkpoint.
To Reproduce
Here are some code snippets from my project.
Main:
    checkpoint_callback = ModelCheckpoint(monitor='val_acc', save_top_k=1, mode='max')

    # Create PTL trainer
    trainer = Trainer.from_argparse_args(Namespace(**dict(train_config)),
                                         default_root_dir=os.getcwd()
                                                           + '/'
                                                           + config["project"]["models"]
                                                           + experiment_name,
                                         checkpoint_callback=checkpoint_callback)

    # Create model from checkpoint
    if model_config["checkpoint"]["load"]:
        model = globals()[hparams["model"]["name"]].load_from_checkpoint(model_config["checkpoint"]["path"])
        trainer = Trainer(resume_from_checkpoint=model_config["checkpoint"]["path"])

        # Evaluate test set from loaded model
        if model_config["test_only"]:
            trainer.test(model)
            exit()

    # Train from scratch
    else:
        model = globals()[hparams["model"]["name"]](Namespace(**dict(hparams)))

    # Train and test with best model
    trainer.fit(model)
    trainer.test(ckpt_path='best')

Validation and test steps from the model:
    def validation_step(self, batch, batch_idx):
        batch, y = batch
        y_hat = self.forward(batch)

        loss = F.cross_entropy(y_hat, y.long())
        labels_hat = torch.argmax(y_hat, dim=1)
        n_correct_pred = torch.sum(y == labels_hat).item()

        return {'val_loss': loss, "n_correct_pred": n_correct_pred, "n_pred": len(y)}

    def validation_epoch_end(self, outputs):
        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        val_acc = torch.tensor(sum([x['n_correct_pred'] for x in outputs]) / sum(x['n_pred'] for x in outputs))
        tensorboard_logs = {'val_loss': avg_loss, 'val_acc': val_acc, 'step': self.current_epoch}

        return {'log': tensorboard_logs}

    def test_step(self, batch, batch_idx):
        batch, y = batch
        y_hat = self.forward(batch)

        loss = F.cross_entropy(y_hat, y.long())
        labels_hat = torch.argmax(y_hat, dim=1)
        n_correct_pred = torch.sum(y == labels_hat).item()

        return {'test_loss': loss, "n_correct_pred": n_correct_pred, "n_pred": len(y)}

    def test_epoch_end(self, outputs):
        avg_loss = torch.stack([x['test_loss'] for x in outputs]).mean()
        test_acc = sum([x['n_correct_pred'] for x in outputs]) / sum(x['n_pred'] for x in outputs)
        tensorboard_logs = {'test_loss': avg_loss, 'test_acc': test_acc, 'step': self.current_epoch}

        return {'log': tensorboard_logs}

Environment

PyTorch Version: 1.5.1
PyTorch Lightning version: 0.8.5
OS: Linux
How you installed PyTorch : pip
Python version: 3.6.9
CUDA/cuDNN version: 10.2
GPU models and configuration: GeForce GTX 860M

Additional context
I am using 16 bits precision but the same bugs appear when using 32 bits.
Thanks'!

@adeboissiere is this still an actual issue, mind test master? 🦝
otherwise could you please share a complete example to reproduce...

This is likely because you are not turning off dropout. If you use the class based dropout you need to let the layer know when it is in training or not


Our config files are considered like url by the load_yaml_from_url_or_path method in util.py

Hi,
After the installation of TPV on our fr instance, We encountered this issue :
 File "/shared/mfs/data/dev/galaxy/server/lib/galaxy/jobs/mapper.py", line 236, in __cache_job_destination
   params, raw_job_destination=raw_job_destination
 File "/shared/mfs/data/dev/galaxy/server/lib/galaxy/jobs/mapper.py", line 223, in __determine_job_destination
   job_destination = self.__handle_dynamic_job_destination(raw_job_destination)
 File "/shared/mfs/data/dev/galaxy/server/lib/galaxy/jobs/mapper.py", line 203, in __handle_dynamic_job_destination
   return self.__handle_rule(expand_function, destination)
 File "/shared/mfs/data/dev/galaxy/server/lib/galaxy/jobs/mapper.py", line 206, in __handle_rule
   job_destination = self.__invoke_expand_function(rule_function, destination)
 File "/shared/mfs/data/dev/galaxy/server/lib/galaxy/jobs/mapper.py", line 123, in __invoke_expand_function
   return expand_function(**actual_args)
 File "/shared/mfs/data/dev/galaxy/.venv/lib/python3.7/site-packages/tpv/rules/gateway.py", line 50, in map_tool_to_destination
   ACTIVE_DESTINATION_MAPPER = setup_destination_mapper(app, tpv_config_files)
 File "/shared/mfs/data/dev/galaxy/.venv/lib/python3.7/site-packages/tpv/rules/gateway.py", line 28, in setup_destination_mapper
   mapper = load_destination_mapper(tpv_config_files)
 File "/shared/mfs/data/dev/galaxy/.venv/lib/python3.7/site-packages/tpv/rules/gateway.py", line 19, in load_destination_mapper
   current_loader = TPVConfigLoader.from_url_or_path(tpv_config_file)
 File "/shared/mfs/data/dev/galaxy/.venv/lib/python3.7/site-packages/tpv/core/loader.py", line 128, in from_url_or_path
   tpv_config = util.load_yaml_from_url_or_path(url_or_path)
 File "/shared/mfs/data/dev/galaxy/.venv/lib/python3.7/site-packages/tpv/core/util.py", line 13, in load_yaml_from_url_or_path
   else:
 File "/shared/mfs/data/dev/galaxy/.venv/lib/python3.7/site-packages/requests/api.py", line 73, in get
   return request("get", url, params=params, **kwargs)
 File "/shared/mfs/data/dev/galaxy/.venv/lib/python3.7/site-packages/requests/api.py", line 59, in request
   return session.request(method=method, url=url, **kwargs)
 File "/shared/mfs/data/dev/galaxy/.venv/lib/python3.7/site-packages/requests/sessions.py", line 573, in request
   prep = self.prepare_request(req)
 File "/shared/mfs/data/dev/galaxy/.venv/lib/python3.7/site-packages/requests/sessions.py", line 496, in prepare_request
   hooks=merge_hooks(request.hooks, self.hooks),
 File "/shared/mfs/data/dev/galaxy/.venv/lib/python3.7/site-packages/requests/models.py", line 368, in prepare
   self.prepare_url(url, params)
 File "/shared/mfs/data/dev/galaxy/.venv/lib/python3.7/site-packages/requests/models.py", line 440, in prepare_url
   f"Invalid URL {url!r}: No scheme supplied. "
requests.exceptions.MissingSchema: Invalid URL '/shared/mfs/data/dev/galaxy/config/total_perspective_vortex/users.yml': No scheme supplied. Perhaps you meant https:///shared/mfs/data/dev/galaxy/config/total_perspective_vortex/users.yml?

I was the same error when I ran the dry-run cli commands if I don't add explicitly yaml files
I have not ideas how I could change the situation. Also, inside a venv, I don't know how I could debug to display variables content
Thanks by advance,

It's because I don't fill users.yml, sorry for this issue too quickly

Glad to know you found the issue, will close.


Error when creating SSL keys for DS

When I try to create SSL keys for a deliveryservice I am getting the following error:

SSL keys for 'deliveryservice' could not be created.  Response was: Error creating key and csr. Result is -1

This was confirmed on 1.5.x and master.

This issue happens when keys are first PASTED for a DeliveryService (never generated) and then you try to generate keys.
When the add keys screen is accessed, Traffic Ops grabs the existing keys from Riak, then when you attempt to generate keys, Traffic Ops uses the hostname from the existing keys record.  Since the keys were originally pasted and not generated, a hostname is not stored with the keys causing an error when trying to generate new keys.  Traffic Ops should be updated to allow the user to enter the hostname OR get the hostname using the DS regex (like it does when generating keys).

The issue above is also present in previous version of Traffic Ops, one difference is that it results in a reponse of

SSL keys for 'neuman-dns' could not be created.  Response was Error creating key and csr. Result is 256

The original issue was due to using an invalid riak config.  The second issue is present in current and previous version of Traffic Ops, so assigning to 1.6.


how set text gravity center?

how to set text gravity center ?

after declaring params write this:
params.gravity = Gravity.TOP;


Why do I need this?

It isn't immediately clear from the front page, why I need this, or why it would be helpful to my workflow. Could you elaborate on what this extension is, how it could be used, etc? It has a one sentence description, which is very vague, and then hits ya with a ton of setup and install info.
Please help. :P

You don't.
From what I could immediately gather, it does more or less what the 'styles' dropdown does, just with an image on top. It looks interesting but writing your own config files seems too much work for what you get.

well basically I use it to find good style combinations with different characters and models.
The configuration is actually pretty easy, and the result not bad.
Imo it is similar to prompt matrix, but easier to search through


chore(ci): add 32-bit alignment check

add a CI check that makes use of SA1027 from staticcheck which correctly detects alignment bugs when compiled for GOARCH=i386 (it doesn't find it when built under x86_64/arm64)
align suppressedHighWaterMarkOffset to fix the issue reported in #2873

Fixes #2873

@twmb fyi, you might be interested in the CI check as its a faster/simpler version of what you did here (and removed here due to slowness) as it doesn't need a full qemu boot

Ty for the callout! I worked it into https://github.com/twmb/franz-go/pull/715. This is way better than my bad junk I eventually ripped out :)


Buttons

When I select different parts of a document, the bold, italic, underline, bullet and alignment buttons do not change to suit it.

This has been fixed.


[#7]: Categories Screen

[#7 ]This PR will integrate a categories screen and a list of categories of quotes.

Closed [#7 ]


enhancement split delete bitmap from tablet meta

Proposed changes
Issue Number: close #xxx

In MoW table，save meta will be called when publish. It will serialize rowset meta, delete bitmap and other tablet meta info. The size of the tablet meta is relatively large, and persistence will cause performance overhead to Rocksdb.
So I split delete bitmap form tablet meta, only save delete bitmap when publish. Regularly perform checkpoint to save delete bitmap to tablet meta.


Polymorphic Encoding

This WIP Draft makes the Encoder trait polymorphic also on the representation type of Json.
circe is built around the purely-functional Json data type, which represents syntax trees in the Json **grammar. An Encoding is just a function from a data type A to a Json value.  circe then provides many implicit standard encoders for base data types, and utilities to derive encoders generically.
Having a representation of Json as a AST data structure helps to query it, or modify it in some ways. However, an application may only use Json to generate an HTTP response. In the case of http4s with circe, the application may encodes an internal response into a Json object, only to immediately encode that Json value into a Chunk[Byte] that goes in the response. code. In this case, it may be more efficient to directly encode the A value into the chuk, by-passing the Json value.
One solution to this problem, of course, is to use a library like jsoniter, that directly generate the bytearray. However, this functionality can be incorporated into circe itself.
We can do this by making Encoder[A] polymorphic on the target representation, so we may directly define encoders from base types. Notably, most of the encoders provided by circe, as well as the derivation, are not tied to to the Json data type itself: they work by mapping or contramapping other encoders.

Thanks for this proposal. My current plan is to publish a 1.0 release using the current model at some point over the next few months, and then to immediately start 2.0 series milestones to explore less AST-dependent approaches for both encoding and decoding. I'm not sure what that will look like, but I'm not convinced that the encoder or decoder type classes should be parameterised on the representation. In any case it's useful to have this as a concrete starting point on the encoder side to evaluate.


Uaa server fixes

big thanks! I am currently stucked at work at the moment, so this is really helping me!


V1.7.1: Module not found: Error: Can't resolve 'feathers-hooks-common/lib/populate' in '/app/node_modules/feathers-hooks/lib' @ ./~/feathers-hooks/lib/hooks.js 13:16-61

Steps to reproduce
I don't know how to describe about it, but i wonder why them have the strange relation.
I built it with angular2 hot module style.
Module Loader:
"webpack": "2.2.0",
"webpack-dev-middleware": "^1.10.1",
"webpack-dev-server": "2.2.0",
"webpack-dll-bundles-plugin": "1.0.0-beta.5",

Can you share a repository to reproduce this error? feathers-hooks-common populate should not be used or imported on the client.

So, sorry, i tried to reproduce, but it did not occur today, maybe some options wrong with my webpack config. i will close this issues, if i meet it again, i will reopen it.
thank you, @daffl

Getting this error as well:
ERROR in ./~/feathers-hooks/lib/hooks.js
Module not found: Error: Can't resolve 'feathers-hooks-common/lib/populate' in '/Users/david/Data/git/node/cabal/node_modules/feathers-hooks/lib'
 @ ./~/feathers-hooks/lib/hooks.js 13:16-61
 @ ./~/babel-loader/lib?{"presets":["vue-app"],"cacheDirectory":true}!./~/vue-loader/lib/selector.js?type=script&index=0!./pages/auth/login.vue
 @ ./pages/auth/login.vue
 @ ./.nuxt/router.js
 @ ./.nuxt/index.js
 @ ./.nuxt/client.js
 @ multi webpack-hot-middleware/client?reload=true ./.nuxt/client.js
ERROR in ./~/feathers-hooks/lib/hooks.js
Module not found: Error: Can't resolve 'feathers-hooks-common/lib/bundled' in '/Users/david/Data/git/node/cabal/node_modules/feathers-hooks/lib'
 @ ./~/feathers-hooks/lib/hooks.js 15:15-59
 @ ./~/babel-loader/lib?{"presets":["vue-app"],"cacheDirectory":true}!./~/vue-loader/lib/selector.js?type=script&index=0!./pages/auth/login.vue
 @ ./pages/auth/login.vue
 @ ./.nuxt/router.js
 @ ./.nuxt/index.js
 @ ./.nuxt/client.js
 @ multi webpack-hot-middleware/client?reload=true ./.nuxt/client.js

All hooks and utilities are now available off feathers-hooks-common
directly.
On Thu, Apr 13, 2017 at 3:17 PM, bdschr<EMAIL_ADDRESS>wrote:

Getting this error as well:
ERROR in ./~/feathers-hooks/lib/hooks.js
Module not found: Error: Can't resolve 'feathers-hooks-common/lib/populate' in '/Users/david/Data/git/node/cabal/node_modules/feathers-hooks/lib'
@ ./~/feathers-hooks/lib/hooks.js 13:16-61
@ ./~/babel-loader/lib?{"presets":["vue-app"],"cacheDirectory":true}!./~/vue-loader/lib/selector.js?type=script&index=0!./pages/auth/login.vue
@ ./pages/auth/login.vue
@ ./.nuxt/router.js
@ ./.nuxt/index.js
@ ./.nuxt/client.js
@ multi webpack-hot-middleware/client?reload=true ./.nuxt/client.js
ERROR in ./~/feathers-hooks/lib/hooks.js
Module not found: Error: Can't resolve 'feathers-hooks-common/lib/bundled' in '/Users/david/Data/git/node/cabal/node_modules/feathers-hooks/lib'
@ ./~/feathers-hooks/lib/hooks.js 15:15-59
@ ./~/babel-loader/lib?{"presets":["vue-app"],"cacheDirectory":true}!./~/vue-loader/lib/selector.js?type=script&index=0!./pages/auth/login.vue
@ ./pages/auth/login.vue
@ ./.nuxt/router.js
@ ./.nuxt/index.js
@ ./.nuxt/client.js
@ multi webpack-hot-middleware/client?reload=true ./.nuxt/client.js
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/feathersjs/feathers-hooks/issues/142#issuecomment-293996387,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABezn1p4bQ3J0Wais_5rs5dHtWpsbl2Uks5rvnTRgaJpZM4MTXej
.

Thanks for the swift answer. Am new to feather.js and until now only used node for compiling es6 vue.js components. But can you elaborate exactly what's up with feathers-hooks-common? I tried to use the following vue.js (nuxt.js) component when I ended up with the error above.
<script>
import feathers from 'feathers/client'
import rest from 'feathers-rest/client'
import auth from 'feathers-authentication-client'
import hooks from 'feathers-hooks'
import axios from 'axios'
import localStorage from 'localstorage-memory'

const host = 'http://localhost:3030'
const app = feathers().configure(hooks())
			.configure(rest(host).axios(axios))
			.configure(auth({ storage: localStorage }))

const userService = app.service('users')

export default {
	layout: 'auth',
    data() {
	     return {
	     	loginForm: {
	     		email: '',
	     		password: ''
	     	}
	     }
    },
    methods: {
    	login() {
    		console.log(this.loginForm);
    	}
    }
  }
</script>

When using feathers-hooks-common I get:
TypeError: __WEBPACK_IMPORTED_MODULE_3_feathers_hooks_common___default(...) is not a function
    at Object.exports.modules.Array.concat.Object.defineProperty.value (login.vue:28:0)
    at __webpack_require__ (webpack:/webpack/bootstrap 88ec5359c6ec0ae585f9:25:0)
    at Object.exports.modules.Array.concat.Object.defineProperty.value (pages/auth/login.vue:3:0)
    at __webpack_require__ (webpack:/webpack/bootstrap 88ec5359c6ec0ae585f9:25:0)

Many thanks!

What version of feathers hooks common are you using?  The latest is 3.0.0. All hooks are available off require ('feathers-hooks-common') with no subfolders.
I don't see it being used in the example above.

THis is a duplicate of https://github.com/feathersjs/feathers-hooks/issues/149


Use postfix for() instead of map {} because map is slower than for()

Coverage remained the same at 95.144% when pulling 436bc25f5bd065a89b00a82328c48ca1a6413a2e on v4255p10 into 5d34121a8839d6b24c8c1c22560cf05167f706e6 on master.


Introduce Cancellable API

Context
See #2128
All the time Gradle can't kill the build running in daemon. For example, if we try to press Ctrl-C, Gradle will mark the daemon as Cancelled then wait 10s for the build to finish (do nothing but wait). Unfortunately in most cases 10s is not enough, so you will see a few lines in daemon log:
Cancel: daemon is still busy after grace period. Will force stop.
...
Daemon vm is shutting down...

The consequence is, in most cases, pressing Ctrl-C will kill the daemon.
This PR introduces an interface Cancellable to allow tasks to react to the Ctrl-C. Cancellable tasks' cancel() callbacks will be executed upon cancellation.

Closing as this was superseded

https://github.com/gradle/gradle/pull/5249
A public API will require some more discussion first. Meanwhile the above fixes the most common problem.


Update tenant-management-check-tenant-creation-permission.md

Removing "(preview)" as the feature is now GA

@kengaderdus
Can you review the proposed changes?
When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

#sign-off


update documentation around jquery-rails

This was a good couple of hours trying to chase this down between ember-cli issues and this repository. Seems like it should be documented and also not the default case for Ember CLI apps.
Related https://github.com/rwz/ember-cli-rails/issues/32

@fivetanley Definitely.  This is something we want to make explicit.  Or at least make a way to disable this behavior if necessary.   The perhaps incorrect assumption was that user would include jquery with their rails assets and we'd save space, but we've found several cases where this isn't desired.

@fivetanley I'm not sure that we should advocate that the user remove jquery-rails as it is likely used elsewhere.  I'm going to try to add a flag to the initializer tonight to disable the check for jquery-rails so that ember will include jquery when the flag is set.   Hopefully that'll help

I think the behavior of not including jquery is very confusing, especially as it is the default in ember-cli and also not included in any docs/warnings in the rails console. I feel like excluding jquery from ember should be opt-out and in the documentation.

Im not really sure what you mean. If jquery-rails gem is present, we use that jquery and tell ember-cli to not concat into its vendor.js. Otherwise we would be including jQuery twice.

@rwjblue The scenario that I'm worried about is

User has jquery-rails and uses it in parts of their app (or as we recently found out has it as a dependency of another gem)
User includes their ember app onto a page that doesn't also include applicaton.js

In this scenario since the Jquery constant is found we tell ember not to send it which breaks their app (unexpectedly).


image_dataset.py doesn't work if an input image is gray scale

If the image is gray scale, an error, "ValueError: axes don't match array" will occur in line 123.

Thank you for sending the PR. Could you change the title more explanatory one?

I'm sorry for your inconvenience. I changed the title.

Please, also fix ImageDataset after this line https://github.com/pfnet/chainer/blob/master/chainer/datasets/image_dataset.py#L61 (analogous code).

Could you please write a minimal test demonstrating the fix, using RGB and greyscale data in chainer/chainer/datasets/image_dataset.py?

I also changed docs in examples/imagenet/train_imagenet.py because ImageNet includes some greyscale images.
If we don't convert them to color images, channel size doesn't match with all images.

Great, thank you. LGTM except the above two comments. image_dataset.py is not tested yet, so perhaps it's alright to leave it like that for the moment.

Thank you for your advices. I fixed the code.

Do you have a minimum code to test it?

shape of images are different between colored images and greyscale images. Is it acceptable?

On offline discussion, we concluded that a user is responsible to convert colored/greyscale images.
LGTM.


feat(category,navigation): replace category ID with UID in magento driver

PR Checklist
Please check if your PR fulfills the following requirements:

[x] The commit message follows our guidelines: https://github.com/graycoreio/daffodil/blob/master/CONTRIBUTING.md#commit
[x] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)

PR Type
What kind of change does this PR introduce?

[ ] Bugfix
[x] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] CI related changes
[ ] Documentation content changes
[ ] Other... Please describe:

What is the current behavior?

The category and navigation magento drivers use the deprecated category id.
What is the new behavior?
Those drivers use the new uid field and query the category with category_uid. Many string coercions were removed.
Does this PR introduce a breaking change?
[ ] Yes
[x] No

Other information

The navigation request works, but the category request doesn't seem to.

@lderrickable are you running it in demo? Do you have URL minification enabled for Apollo?

@griest024 Whoops. Sorry, brain went smooth. lgtm


PGP keygen - add copy and make pk selectable

r? @keybase/react-hackers

@keybase/react-hackers bump!


could you have an armhf image?

docker load the image on armf but refuse executing: this is because the image is not compiled for armf...

Hi @gbrault, can you elaborate what is armf? I searched on Google but didn't found concrete results. You mean generate a binary release for arm processors?

I found this docker user . Are you talking about generate a DPS docker image based on images of that user repository like https://hub.docker.com/r/arm64v8/debian/ ?

If yes, which arm are you interested in? Is this image ok for you ? https://hub.docker.com/r/arm64v8/debian/

Thanks @mageddo, I elaborate:
1- I have to investigate the reference you provided
2- The reference image for me for ARM is Debian based raspberry pi aka raspbian https://www.raspberrypi.org/downloads/raspbian/
3- I misspelled armf actually it's armhf (armhf stands for "arm hard float", and is the name given to a debian port for arm processors (armv7+) that have hardware floating point support), Anyway Raspbian is the target (you will have to figure out: maybe they are 32 bits and 64 bits flavor). My target processor is Raspberry pi 3 (which is 64 bits?) ( uname -a => Linux raspberrypi 4.14.69-v7+ #1141 SMP Mon Sep 10 15:26:29 BST 2018 armv7l GNU/Linux )

Okay, I compiled arm and arm64 images, you could try to run them
dns-proxy-server-linux-arm-2.7.0.tgz
dns-proxy-server-linux-arm64-2.7.0.tgz
If you need the docker docker image

Clone this branch
Download dns-proxy-server-linux-arm-2.7.0.tgz and dns-proxy-server-linux-arm64-2.7.0.tgz then move them on /build directory (create this if necessary)
Build docker image using

$ docker-compose build prod-build-arm7x86-image-dps
$ docker-compose build prod-build-arm8x86-image-dps
$ docker-compose build prod-build-arm8x64-image-dps

Thanks: I give a try and let you know!

I am not successful building the docker image (32 bits).
1- cloned https://github.com/mageddo/dns-proxy-server/tree/feature/87
2- created build directory
3- copied both file in build
4- docker-compose build prod-build-arm7x86-image-dps
did that on a windows10 machine
tells me
C:\Users\gbrau\dns-proxy-server>docker-compose build prod-build-arm7x86-image-dps
WARNING: The PWD variable is not set. Defaulting to a blank string.
WARNING: The TRAVIS_BRANCH variable is not set. Defaulting to a blank string.
WARNING: The REPO_TOKEN variable is not set. Defaulting to a blank string.
WARNING: The HOME variable is not set. Defaulting to a blank string.
ERROR: No such service: prod-build-arm7x86-image-dps
Any idea?

Look likes prod-build-arm7x86-image-dps it's no present on docker-compose.yml, but it is for me. Make sure this is our last commit, try
$ git checkout feature/87

or
git checkout b714a857f46afe69e5889b6b4b8d3192eff99506

Then try it again

Also make sure you are running docker-compose on folder that have that content
.
├── build
├── builder
├── cache
├── conf
├── controller
├── countlines
├── dns.go
├── docker-compose.yml
├── Dockerfile
├── Dockerfile.arm7x86
├── Dockerfile.arm8x64
├── Dockerfile.arm8x86
├── Dockerfile.hub
├── docs
├── events
├── files
├── flags
├── LICENSE
├── log
├── proxy
├── README.md
├── RELEASE-NOTES.md
├── resolvconf
├── service
├── static
├── test
├── utils
├── vendor
├── VERSION

Ok, the container is now in my Docker HUB
you can see https://hub.docker.com/r/gbrault/dns-proxy-server-arm7x86/
I have it alive in my RPI up and running!
I need now to test it!
I let you know the result

Most of it works, but I have an issue when not providing --hostnames creating containers: it seems you then use the container ID as the default name. Here is an excerpt of the logs
2018/10/13 09:07:53 INFO f=docker.go:60 pkg=github.com/mageddo/dns-proxy-server/events/docker m=HandleDockerEvents status=container-from-list-begin, container=/nextcloud
2018/10/13 09:07:53 INFO f=local.go:34 pkg=github.com/mageddo/dns-proxy-server/events/local m=LoadConfiguration status=begin, confPath=/app/conf/config.json
2018/10/13 09:07:53 INFO f=local.go:37 pkg=github.com/mageddo/dns-proxy-server/events/local m=LoadConfiguration  status=openingFile
2018/10/13 09:07:53 INFO f=local.go:65 pkg=github.com/mageddo/dns-proxy-server/events/local m=LoadConfiguration  status=success
2018/10/13 09:07:53 DEBUG f=docker.go:173 pkg=github.com/mageddo/dns-proxy-server/events/docker m=putHostnames container=/nextcloud, network=myNetwork, ip=<IP_ADDRESS>
2018/10/13 09:07:53 DEBUG f=docker.go:185 pkg=github.com/mageddo/dns-proxy-server/events/docker m=putHostnames host=5b77a6c81d52, ip=<IP_ADDRESS>
2018/10/13 09:07:53 INFO f=docker.go:66 pkg=github.com/mageddo/dns-proxy-server/events/docker m=HandleDockerEvents status=container-from-list-success, container=/nextcloud, hostnames=[5b77a6c81d52]
So I have a question: when hostnames is not specified what is the default?
Locking to docker documentation => please see
Container networking
Attached complete log
log.txt

I have found the issue: portainer, the deployment tool I use is setting this hostname...
So: no need for investigation!
Thx, I keep you posted

So I have a question: when hostnames is not specified what is the default?

DPS uses HOSTNAMES env and/or --hostnames option. I'm not sure but reading the link you posted and based on my past experiences, docker will put the container name at hostname option then DPS will use that..
DPS has a new feature when you can set container name / service name as hostname, but it is disabled by default see the reference

Glad it works for you, so next step is to officially generate ARM images, I think it would be great to document how compile DPS for others platforms too.

I'm just curious, What is the objective to you run docker inside a RPI with portainer, DPS, etc. ?

Glad it works for you, so next step is to officially generate ARM images, I think it would be great to document how compile DPS for others platforms too.

Indeed - any news/plans on the officially generated ARM images?

@hgross I already did some code but didn't complete, I'm working on this now and  I plan arm support to be released on the next DPS version, not sure when cause I code on my free time.

Awesome, just tested it on a Raspberry Pi 3 B+ and it works as expected.
Regarding Gradle - I can only recommend it.
We use it a lot (also with docker, check out https://github.com/palantir/gradle-docker) and are happy with it.

Ok, I'll mark it as done. If you have some issue feel free to start a talk


Percona XtraDB CREATE TABLES AS SELECT prohibited

We're using Flyway 6.4.4
We are using Percona XtraDB Cluster 5.7. The pxc_strict_mode is set to MASTER, which in our case is hard coded in the bosh-release we are using (https://github.com/cloudfoundry-incubator/pxc-release).
When trying to 'migrate' we see the following error:
Migration  failed
-----------------
SQL State  : HY000
Error Code : 1105
Message    : Percona-XtraDB-Cluster prohibits use of CREATE TABLE AS SELECT with pxc_strict_mode = ENFORCING or MASTER
Location   :  ()
Line       : 1

Flyway already has a check for the pxc_strict_mode, which is then also used to decide whether "create table as select" is allowed. However, the pxc_strict_mode check only checks for "ENFORCING" and ignores "MASTER". Therefore, in our case the migration is still done by "create table as select", which is also not supported with pxc_strict_mode = MASTER (see error message above).
I think the pxc_strict_mode check, could also look for MASTER.

Will be fixed in 6.5.2 - thankyou!


The "Reply URL (Assertion Consumer Service URL)" can not be empty

The "Reply URL (Assertion Consumer Service URL)" is revealed only if you check the "Show Advanced URL Settings". Initially it is empty.
If you leave that "Reply URL" empty then when you try to connect you redirected to "https://yoursharepointserverurl/" with a "Server not found error".

To make it work you have to set the same value as the "Sign on URL".
Check the following image.

Document details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: abc120a6-1b98-874a-1cfb-b43456f4a718
Version Independent ID: 1008851f-9a43-953c-e6c1-269397268c72
Content: Tutorial: Azure Active Directory integration with SharePoint on-premises
Content Source: articles/active-directory/saas-apps/sharepoint-on-premises-tutorial.md
Service: active-directory
GitHub Login: @jeevansd
Microsoft Alias: jeedes

@tbebekis Thanks for the feedback !  I have assigned this issue to content author to investigate and update the document as appropriate.

@tbebekis I agree with the issue here. This is why now we have made the Reply URL field as mandatory so that you can specify the correct value in the field.

@chetansriv I have now made the Reply URL mandatory here can you please update the doc?

Tutorial updated
#please-close


Execute .SQL files

I want to be able to execute full .SQL files by passing them through the driver. How can i do that?

Hi @maddy2u ,
The driver does not support that functionality at this time, so some pre-proccessing would need to occur in your script before passing the individual statements to the driver. Something like the code snippit below could work:
import redshift_connector
import os

MY_FILE: str = os.environ.get('SQL_SMTS', None)

def preprocess_sql_file(filepath: str) -> list[str]:
   """Returns a list of individual SQL statements"""
  raise NotImplementedError

with redshift_connector.connect(...) as conn:
   with conn.cursor() as cursor:
      for stmt in preprocess_sql_file(MY_FILE):
         cursor.execute(stmt)
   conn.commit() # optionally, commit changes

I will chat with the team to see if we can add support for executing .SQL files directly and update this issue with my findings.

Thanks. Any update on the above. Would be a good functionality to have inbuilt.

Hey @maddy2u ,
To support executing .SQL files directly we would need to re-work how the driver sends statements to the Amazon Redshift server. Currently, prepared statements are used in all cases-- which prevents us from supporting mult-statements such as:
cursor.execute("SELECT 1; SELECT 2; SELECT 3;")

this also prevents us from supporting .SQL file execution. We would need to move from using prepared statements to statements to support this.
I'm working to gauge how much re-work this change would require and get an idea about how this would impact the drivers performance.

Makes sense. Looking forward to your response @Brooke-white ! Thank you.

Hi @maddy2u ,
At this time we do not plan to add functionality to execute .SQL files using redshift_connector. Parsing SQL is a big undertaking -- enough so that an entire project, sqlparse, has been dedicated to it.
We are continuing to scope work for supporting multi-statement execution.
Here's an example of how sqlparse could be used for executing .SQL files with redshift_connector. Please see the documentation for sqlparse here for more details.
import redshift_connector
import sqlparse

with open('data.sql', 'r') as file:
    raw_sql_file = file.read()
    sql_stmts = sqlparse.split(raw_sql_file)

    with redshift_connector.connect(...) as conn:
        with conn.cursor() as cursor:
            for stmt in sql_stmts:
                cursor.execute(stmt)

@Brooke-white thanks for advising a workaround. It'll be nice to support multiple statements in a single execute call. I notice psycopg allow me to do this, but I was not able to identify exactly which line from its source code provides such support.

HI @liuming ,
No problem -- yes I agree it'd be much cleaner (and simpler) to support this in a single call to execute.
I believe psycopg supports this due to the approach taken to executing the statement. There are two approaches a driver can take to execute a statement -- simple query cycle and  extended query cycle. Simple query cycle supports executing multiple statements, while extended query protocol does not. redshift_connector uses extended query protocol, which is the root cause for our lack of support. If you're interested, the above links give an interesting overview of the two approaches and their trade offs.
When I have extra time I am working to add support for simple query cycle (i.e. multiple statement execution), but there's no timeline on when this would be available as we are prioritizing other tasks at this time. Ideally, I'd like to provide support for both extended and simple query cycles so users have the option to fine tune how statements are executed :)

@Brooke-white I really appreciate you took the time to link these documentations and also provide a surprising detailed reply. I learn something new today. Thank you. :clap:

closing this out as we won't be supporting direct execution of SQL files, or multi-statements in the short term


SCIM issue- PATCH/ Remove Group Member does not work as expected

Affected - wso2is 5.10 m9
Severity: Medium
Priority: Medium
Description
Even response return 200, user still has the removed role.
curl -v -k --user admin:admin -X PATCH -d '{"schemas":["urn:ietf:params:scim:api:messages:2.0:PatchOp"],"Operations":[{"op":"remove","path":"members[display eq kim,value eq 81ffe274-19a9-46aa-acfe-469a09cdd25a]"}]}' --header "Content-Type:application/json" https://localhost:9443/scim2/Groups/1b91c4ef-9834-4278-82e3-ed810a94403b
WARN log-
WARN {org.wso2.carbon.identity.scim2.common.impl.SCIMUserManager} - There is no updated field in the group: PRIMARY/manager. Therefore ignoring the provisioning.

@NilukaSripalim Was this request obtained from our docs?

@mefarazath
Yes, I can find it here,
https://is.docs.wso2.com/en/next/develop/using-the-scim-2.0-rest-apis/#patch-remove-group-member

@senthalan Let's fix the docs then since we only support simple filters.

Reopening this issue because  issue is observe in  product version 5.10.-alpha 3. [1]
[1] https://github.com/wso2/product-is/releases/download/v5.10.0-alpha3/wso2is-5.10.0-alpha3.zip

Tried with the 5.10.0-Aplha3. Could not reproduce the issue with uniqueIdJDBC and UniqueIDReadWriteLdap user stores.
I am resolving the issue. Please reopen if you still face the issue. Please provide the steps to recreate the issue.

Issue reopen since it reproduce in product version [1]. SCIM2 document [2]
[1] https://github.com/wso2/product-is/releases/download/v5.10.0-alpha3/wso2is-5.10.0-alpha3.zip
[2] https://is.docs.wso2.com/en/next/develop/using-the-scim-2.0-rest-apis/#patch-remove-group-member
Steps:
01. Create user:
request:
curl -v -k --user admin:admin --data '{"schemas":[],"name":{"familyName":"jackson","givenName":"kim"},"userName":"kim","password":"kimwso2","emails":[{"primary":true,"value":"kim.jackson@gmail.com","type":"home"},{"value":"kim_j@wso2.com","type":"work"}]}' --header "Content-Type:application/json" https://localhost:9443/scim2/Users
response:
{"emails":[{"type":"work","value":"kim_j@wso2.com"},{"type":"home","value":"kim.jackson@gmail.com"}],"meta":{"created":"2020-02-05T17:57:38.128Z","location":"https://localhost:9443/scim2/Users/59116eda-24fd-4305-af5a-e7835187a53b","lastModified":"2020-02-05T17:57:38.128Z","resourceType":"User"},"schemas":["urn:ietf:params:scim:schemas:core:2.0:User","urn:ietf:params:scim:schemas:extension:enterprise:2.0:User"],"roles":[{"type":"default","value":"Internal/everyone"}],"name":{"givenName":"kim","familyName":"jackson"},"id":"59116eda-24fd-4305-af5a-e7835187a53b","userName":"kim"}
02. Create group:
request:
curl -v -k --user admin:admin --data '{"displayName":"manager"}' --header "Content-Type:application/json" https://localhost:9443/scim2/Groups
response:
{"displayName":"PRIMARY/manager","meta":{"created":"2020-02-05T17:58:37.832Z","location":"https://localhost:9443/scim2/Groups/4c498c36-ebc6-4dcd-b3ac-4e338551396d","lastModified":"2020-02-05T17:58:37.832Z","resourceType":"Group"},"schemas":["urn:ietf:params:scim:schemas:core:2.0:Group"],"id":"4c498c36-ebc6-4dcd-b3ac-4e338551396d"}
03. Add member to the group
request:
curl -v -k --user admin:admin -X PATCH -d '{"schemas":["urn:ietf:params:scim:api:messages:2.0:PatchOp"],"Operations":[{"op":"add","value":{"members":[{"display": "kim","$ref":"https://localhost:9443/scim2/Users/59116eda-24fd-4305-af5a-e7835187a53b","value": "59116eda-24fd-4305-af5a-e7835187a53b"}]}}]}' --header "Content-Type:application/json" https://localhost:9443/scim2/Groups/4c498c36-ebc6-4dcd-b3ac-4e338551396d
response:
{"displayName":"manager","meta":{"created":"2020-02-05T17:58:37.832Z","location":"https://localhost:9443/scim2/Groups/4c498c36-ebc6-4dcd-b3ac-4e338551396d","lastModified":"2020-02-05T17:58:37.832Z"},"schemas":["urn:ietf:params:scim:schemas:core:2.0:Group"],"members":[{"display":"kim","value":"59116eda-24fd-4305-af5a-e7835187a53b","$ref":"https://localhost:9443/scim2/Users/59116eda-24fd-4305-af5a-e7835187a53b"}],"id":"4c498c36-ebc6-4dcd-b3ac-4e338551396d"}
04. Get user detail: (to verify user is add to the group)
request:
curl -v -k --user admin:admin 'https://localhost:9443/scim2/Users/59116eda-24fd-4305-af5a-e7835187a53b'
response:
{"emails":[{"type":"work","value":"kim_j@wso2.com"},{"type":"home","value":"kim.jackson@gmail.com"}],"meta":{"created":"2020-02-05T17:57:38.128Z","location":"https://localhost:9443/scim2/Users/59116eda-24fd-4305-af5a-e7835187a53b","lastModified":"2020-02-05T17:57:38.128Z","resourceType":"User"},"schemas":["urn:ietf:params:scim:schemas:core:2.0:User","urn:ietf:params:scim:schemas:extension:enterprise:2.0:User"],"roles":[{"type":"default","value":"manager,Internal/everyone"}],"name":{"givenName":"kim","familyName":"jackson"},"groups":[{"display":"manager","value":"4c498c36-ebc6-4dcd-b3ac-4e338551396d","$ref":"https://localhost:9443/scim2/Groups/4c498c36-ebc6-4dcd-b3ac-4e338551396d"}],"id":"59116eda-24fd-4305-af5a-e7835187a53b","userName":"kim"}
05. Remove user:
request:
curl -v -k --user admin:admin -X PATCH -d '{"schemas":["urn:ietf:params:scim:api:messages:2.0:PatchOp"],"Operations":[{"op":"remove","path":"members[display eq kim,value eq 59116eda-24fd-4305-af5a-e7835187a53b]"}]}' --header "Content-Type:application/json" https://localhost:9443/scim2/Groups/4c498c36-ebc6-4dcd-b3ac-4e338551396d
response:
{"displayName":"PRIMARY/manager","meta":{"created":"2020-02-05T17:58:37.832Z","location":"https://localhost:9443/scim2/Groups/4c498c36-ebc6-4dcd-b3ac-4e338551396d","lastModified":"2020-02-05T17:58:37.832Z"},"schemas":["urn:ietf:params:scim:schemas:core:2.0:Group"],"members":[{"display":"kim","value":"59116eda-24fd-4305-af5a-e7835187a53b","$ref":"https://localhost:9443/scim2/Users/59116eda-24fd-4305-af5a-e7835187a53b"}],"id":"4c498c36-ebc6-4dcd-b3ac-4e338551396d"}
06. Get user detail: (to verify user is remove from the group)
request:
curl -v -k --user admin:admin 'https://localhost:9443/scim2/Users/59116eda-24fd-4305-af5a-e7835187a53b'
response:
{"emails":[{"type":"work","value":"kim_j@wso2.com"},{"type":"home","value":"kim.jackson@gmail.com"}],"meta":{"created":"2020-02-05T17:57:38.128Z","location":"https://localhost:9443/scim2/Users/59116eda-24fd-4305-af5a-e7835187a53b","lastModified":"2020-02-05T17:57:38.128Z","resourceType":"User"},"schemas":["urn:ietf:params:scim:schemas:core:2.0:User","urn:ietf:params:scim:schemas:extension:enterprise:2.0:User"],"roles":[{"type":"default","value":"manager,Internal/everyone"}],"name":{"givenName":"kim","familyName":"jackson"},"groups":[{"display":"manager","value":"4c498c36-ebc6-4dcd-b3ac-4e338551396d","$ref":"https://localhost:9443/scim2/Groups/4c498c36-ebc6-4dcd-b3ac-4e338551396d"}],"id":"59116eda-24fd-4305-af5a-e7835187a53b","userName":"kim"}
User is still assign to the removed group.
Check the console as well

Issue is fixed but need to update the document as well. Therefore close the issue and open a document issue [1]
[1] https://github.com/wso2/docs-is/issues/1118

@aaujayasena @IsuraD We're using v5.10.0, We're still facing the same issue.
Can you confirm if this issue is fixed in any version of 5.10.0 or 5.11.0


Disable two more failing tests in AppVeyor

These two tests only faily on specific platforms (or are non deterministic) in appveyor. They seem to pass locally.
This change just disables them on AppVeyor for now.

Hi @mjbvz, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!

It looks like you're a Microsoft contributor (Matt Bierner). If you're full-time, we DON'T require a Contribution License Agreement. If you are a vendor, please DO sign the electronic Contribution License Agreement. It will take 2 minutes and there's no faxing! https://cla.microsoft.com.

TTYL, MSBOT;

❤ CHECKMARKS!!!! 🎉🎉🎉


Audio device resets to default when enabling particles

OS: Windows 10
Issue Descriptions: Audio device resets to default when enabling particles
Steps to Reproduce:

Set audio device to any but the default
Play a song
Enable particles visualizer (bottom left corner)
Music starts playing through the default audio device

This is a bug caused by Google Play Music.  When you start the particle visualizer, it seizes control of the audio element from me.  There is nothing I can do about this, it just claims it so my code no longer affects it.


Fix typo in example

Whoops, forgot to check for open issues - looks like #4 found this yesterday!


Guarantee stack property in TypeScript typings

Currently TypeScript thinks that the stack property is string | undefined, although it should be string.

Can you fix the lint issues. See Travis.

@sindresorhus What an oversight :man_facepalming: Sorry!! I will pay more attention next time.


Add sample size entry to list for 32bps streams

First of all, many thanks for developing and pointing me to this FLAC decoder. It helped me to prove a point I made in the flac specification:
https://github.com/ietf-wg-cellar/flac-specification/blob/6abc108bccba9e7a985e8ad3ee2b94f3280c3f73/rfc_backmatter.md?plain=1#L8
Specifically

Choosing a 64-bit signed data type for all arithmetic involving sample values would make sure the possibility for overflow is eliminated

At least, I think flac.tcl proves this. I have zero knowledge of Tcl and hadn't even heard of it before encountering this project, but looking at the project and skimming through the list of 'Tcl features' it seems all calculations here are done with at least 64 bit signed ints. Please let me know if I'm wrong.
Anyway, I've recently integrated a 32 bps encoder/decoder into FLAC and have submitted patches for ffmpeg. These patches were rather large, but for flac.tcl I only had to add the sample size to the correct table, after which everything seems to work correctly. I've checked fq as well and stumbled upon the issue that reading of entropy partitions with 'raw bits' doesn't properly work yet (I might look into that myself), but otherwise adding a sample size entry is enough to get it working. I'll submit a PR for that as well.
Once again: many thanks! Having a decoder (counting fq its actually two) that simply works on 32bps FLAC without any changes because it hasn't been designed to use 32-bit signed ints where possible was very valuable in proving that the implementation is correct.

First of all, many thanks for developing and pointing me to this FLAC decoder. It helped me to prove a point I made in the flac specification:
https://github.com/ietf-wg-cellar/flac-specification/blob/6abc108bccba9e7a985e8ad3ee2b94f3280c3f73/rfc_backmatter.md?plain=1#L8
Specifically

Choosing a 64-bit signed data type for all arithmetic involving sample values would make sure the possibility for overflow is eliminated

At least, I think flac.tcl proves this. I have zero knowledge of Tcl and hadn't even heard of it before encountering this project, but looking at the project and skimming through the list of 'Tcl features' it seems all calculations here are done with at least 64 bit signed ints. Please let me know if I'm wrong.

Hey, glad to hear that! yeah TCL might not be a very suitable languages for this kind of thing, long story short I was working on a FLAC binary template for https://github.com/HexFiend/HexFiend which uses TCL.. and then it kind of derailed a bit :) but turns out TCL has quite nice support for building DSL so was great for prototyping some things that ended up in fq.
Yes I think your correct about 64 bit, read up a bit about it and it seems TCL before 8.5 uses C type long (signed 64 i think?) but 8.5 and later uses arbitrary precision integers.

Anyway, I've recently integrated a 32 bps encoder/decoder into FLAC and have submitted patches for ffmpeg. These patches were rather large, but for flac.tcl I only had to add the sample size to the correct table, after which everything seems to work correctly. I've checked fq as well and stumbled upon the issue that reading of entropy partitions with 'raw bits' doesn't properly work yet (I might look into that myself), but otherwise adding a sample size entry is enough to get it working. I'll submit a PR for that as well.

I remember being a bit surprised that it was not that much code also, but it seems if you want things to be more performant things will get complicated fast.
Not sure i follow about the fq issue, problem with 32 bit unary/zigzag encoded integers? but maybe we can continue about that in the fq PR? and i'm more than happy to help out if needed

Once again: many thanks! Having a decoder (counting fq its actually two) that simply works on 32bps FLAC without any changes because it hasn't been designed to use 32-bit signed ints where possible was very valuable in proving that the implementation is correct.

Feel very honored both the projects have been useful for working on the specification! And had forgot that i added partial 32 bit support for both of them :)

Guess i should add some 32bit test at some point also

BTW saw that there will be talk about CELLAR at NTTW6 (https://docs.google.com/spreadsheets/d/1FdZ1_hVZcZc6eVSlxeJMcaDEYW_KjZ_Ye8XcvcvX9HE/edit#gid=0) will you be attending? Think i will be attending and maybe do a talk about fq.

I knew CELLAR would have a in-person meeting 'near' NTTW6, I didn't know it would be at NTTW6, with an audience. I will probably attend.

Nice, might see you there then!


Automated cherry pick of #10762

Cherry pick of #10762 on release-7.2.
/cc  @julmondragon
NONE

Will try to auto merge this PR once all tests and checks are passing. This might take up to an hour.

Trying to auto merge this PR.

Pull Request successfully merged
SHA: 57e67f04daf658b240e59f8d7210ea8bb00c438d


Tweak link format

Old
Blocks unable to be dragged or fields edited (https://github.com/google/blockly/issues/2089)
The link is marked as code so that github won't automatically fix it up.
New
Blocks unable to be dragged or fields edited (#2089)

This change is

Friendly ping.


call merge implementation from graveler

Codecov Report

Merging #1203 (efa9a9d) into master (a73e4d6) will decrease coverage by 0.02%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master    #1203      +/-   ##
==========================================
- Coverage   47.10%   47.08%   -0.03%
==========================================
  Files         197      197
  Lines       15103    15112       +9
==========================================
  Hits         7115     7115
- Misses       7057     7066       +9
  Partials      931      931

Impacted Files
Coverage Δ

graveler/committed/manager.go
0.00% <0.00%> (ø)

graveler/graveler.go
43.56% <0.00%> (ø)

catalog/mvcc/cataloger_create_branch.go
81.53% <0.00%> (-3.08%)
:arrow_down:

catalog/mvcc/cataloger_create_repository.go
62.96% <0.00%> (+3.70%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a73e4d6...efa9a9d. Read the comment docs.

Codecov Report

Merging #1203 (efa9a9d) into master (a73e4d6) will decrease coverage by 0.02%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master    #1203      +/-   ##
==========================================
- Coverage   47.10%   47.08%   -0.03%
==========================================
  Files         197      197
  Lines       15103    15112       +9
==========================================
  Hits         7115     7115
- Misses       7057     7066       +9
  Partials      931      931

Impacted Files
Coverage Δ

graveler/committed/manager.go
0.00% <0.00%> (ø)

graveler/graveler.go
43.56% <0.00%> (ø)

catalog/mvcc/cataloger_create_branch.go
81.53% <0.00%> (-3.08%)
:arrow_down:

catalog/mvcc/cataloger_create_repository.go
62.96% <0.00%> (+3.70%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a73e4d6...efa9a9d. Read the comment docs.


fix: Added prefix to checks only summary

:tada: This PR is included in version 1.4.5 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


fix: remove memory leak in avltree.cpp

Description of Change

This PR removes a memory leak in avltree.cpp. Two things were done:

the new node is created only if root == nullptr,
added the function deleteAllNodes.

Checklist

[x] Added description of change
[x] Added documentation so that the program is self-explanatory and educational - Doxygen guidelines
[x] Relevant documentation/comments is changed or added
[x] PR title follows semantic commit guidelines
[x] Search previous suggestions before making a new one, as yours may be a duplicate.
[x] I acknowledge that all my contributions will be made under the project's license.

Notes:
Quick fix of a memory leak in avltree.cpp.

Will be merging for now. If there are any objections/things to change, feel free to create another PR. 🙂


How do I apply this patch in this link?

Checklist

[x ] I'm asking a question
[x ] I've looked through the README and FAQ for similar questions
[ x] I've searched the bugtracker for similar questions including closed ones

Question
How do I apply this patch?

WRITE QUESTION HERE
In the link below, one person said they fixed the issue with cookies by a patch. How do i apply that patch? I am using kali l.inux
https://github.com/ytdl-org/youtube-dl/issues/15697

The patch was already merged. The patch no longer applies.


[Ruby] Building an array of hashes from Table is very slow

I'm experiencing performance issues when reading Parquet files in Ruby. I've created a very simple synthetic benchmark: a table with a single float column and 1,000,000 rows. I want to read this data into an array of hashes. Here's my code:
require 'arrow'
require 'parquet'
require 'benchmark'

def read_parquet
  table = Arrow::TableLoader.load('data.parquet', { format: :parquet })
  table.each_record(reuse_record: true).map(&:to_h)
end

Benchmark.bmbm do |x|
  x.report("read_parquet")  { read_parquet }
end

And the report:
$ bundle exec ruby --yjit read.rb
Rehearsal ------------------------------------------------
read_parquet  20.894371   0.129733  21.024104 ( 21.066472)
-------------------------------------- total: 21.024104sec

                   user     system      total        real
read_parquet  21.872816   0.079736  21.952552 ( 21.991875)

I also tested equivalent code in Python:
import pyarrow.parquet as pq
import timeit

def read_parquet():
    table = pq.read_table('data.parquet')
    return table.to_pylist()

time_taken = timeit.timeit(read_parquet, number=10)  # Run 10 times
print(f"Average time: {time_taken / 10:.6f} seconds")

which yields:
$ poetry run python read.py
Average time: 0.610864 seconds

It seems that Ruby version is 30-40x slower than pyarrow. Is there anything I can do to improve the performance here?
For completeness, here's a script that generates the test data (data.parquet file):
require 'arrow'
require 'parquet'
require 'benchmark'

schema = Arrow::Schema.new([Arrow::Field.new("foo", :float)])

data = 1_000_000.times.map { {foo: rand} }
table = Arrow::RecordBatchBuilder.build(schema, data).to_table
table.save('data.parquet', format: :parquet, compression: :uncompressed)

Component(s)
Ruby

In general, you should not convert large data to raw Ruby objects. It's slow as you seen.
But Red Arrow provides an optimized feature that converts an Arrow array to a Ruby array. You can use it by arrow::XXXArray#to_a.
For example, you can get a Hash of Arrays (not an Array of Hashes) with the feature:
def read_parquet
  table = Arrow::TableLoader.load('data.parquet', { format: :parquet })
  data = {}
  table.schema.fields.each do |field|
    data[field.name] = []
  end
  table.each_record_batch do |record_batch|
    record_batch.each_column do |column|
      data[column.name].concat(column.data.to_a)
    end
  end
  data
end

Are you really need an Array of Hashes?

Thank you! Based on your advice I came up with this alternative implementation:
def read_parquet2
  table = Arrow::TableLoader.load('data.parquet', { format: :parquet })
  result = []
  table.each_record_batch do |record_batch|
    result.concat(record_batch['foo'].data.to_a.map { {foo: _1} })
  end
  result
end

Obviously this implementation is specialized to this particular schema. But it's orders of magnitude faster (even faster than pyarrow).
$ bundle exec ruby --yjit read.rb
Rehearsal -------------------------------------------------
read_parquet   17.731458   0.070106  17.801564 ( 17.830338)
read_parquet2   0.218525   0.019901   0.238426 (  0.239175)
--------------------------------------- total: 18.039990sec

                    user     system      total        real
read_parquet   21.314701   0.113192  21.427893 ( 21.462763)
read_parquet2   0.169205   0.013307   0.182512 (  0.183047)

In general, you should not convert large data to raw Ruby objects. It's slow as you seen.

I'm aware there's a certain overhead to converting data to Ruby objects. But it seems to me it's not the bottleneck - certainly creating a million Hashes doesn't take ~20 seconds. When I looked at the code of Record#to_h, I realized calling .to_h on each record separately in my original implementation caused repeatedly fetching the whole column.
# record.rb

def to_h
  attributes = {}
  @container.columns.each do |column|
    attributes[column.name] = column[@index]
  end
  attributes
end

Are you really need an Array of Hashes?

In my use case I want to load data from Parquet file into a database. So I need an Array of Hashes because this is what ActiveRecord's insert_all accepts. I think it would be nice to provide a built-in method for doing this in an optimized way. I filed a separate feature request: https://github.com/apache/arrow/issues/45122.

For the use case, I recommend that you use ADBC: https://arrow.apache.org/adbc/
You can use Apache Arrow data to load them to a database. ADBC converts the given Apache Arrow data internally with optimized implementation.
I want to implement Active Record ADBC adapter but it's not completed yet...: https://github.com/red-data-tools/activerecord-adbc-adapter


Numerical issues in integration tests in Linalg on tensors

Describe the bug
There are some integration tests failed in Linalg on tensors because of numerical issue. This happens on both CPU and GPU backends.
Unfortunately, we can not debug it as before because VMLA does not work with -iree-flow-dispatch-linalg-on-tensors. And it fails on both backends. Luckily, the model file can be much smaller because we can use  -iree-flow-strip-and-splat-constant-variables to strip the flow variables, and the issue can be reproduced with the faked weight model.
To Reproduce
Download keyword_spotting_tests_fake-mobilenet-v2.mlir: https://gist.github.com/hanhanW/dafdd5ae914f866e592e124428467aed

Generate the module in Linalg on buffers: iree-translate --iree-mlir-to-vm-bytecode-module --iree-hal-target-backends=vulkan-spirv ~/keyword_spotting_tests_mobilenet-v2.mlir -o a.vmfb
Generate the module in Linalg on tensors: iree-translate --iree-mlir-to-vm-bytecode-module --iree-hal-target-backends=vulkan-spirv ~/keyword_spotting_tests_mobilenet-v2.mlir -o b.vmfb -iree-flow-dispatch-linalg-on-tensors=true -iree-codegen-spirv-experimental-linalg-on-tensors=true
Run the first module: iree-run-module --module_file=a.vmfb --driver=vulkan --entry_function=call --function_inputs=1x16000xf32=0 and you will see twelve 0.0431243.
Run the second module: iree-run-module --module_file=b.vmfb --driver=vulkan --entry_function=call --function_inputs=1x16000xf32=0 and you will see twelve 0.0787472.

I feel that this is reduction fusion issue in generic op. Narrow down the test case to
module  {
  func @call(%arg0: tensor<1x24x1x4xf32> -> tensor<1x4xf32> attributes {iree.module.export} {
    %0 = mhlo.constant dense<0.000000e+00> : tensor<f32>
    %1 = mhlo.constant dense<3.000000e+00> : tensor<1x24x1x4xf32>
    %2 = mhlo.add %arg0, %1 : tensor<1x24x1x4xf32>
    %3 = "mhlo.reduce"(%2, %0) ( {
    ^bb0(%arg1: tensor<f32>, %arg2: tensor<f32>):  // no predecessors
      %4 = mhlo.add %arg1, %arg2 : tensor<f32>
      "mhlo.return"(%4) : (tensor<f32>) -> ()
    }) {dimensions = dense<[1, 2]> : tensor<2xi64>} : (tensor<1x24x1x4xf32>, tensor<f32>) -> tensor<1x4xf32>
    return %3 : tensor<1x4xf32>
  }
}

Run build-tracy/iree/tools/iree-run-mlir b.mlir --iree-hal-target-backends=vulkan-spirv --function-input=1x24x1x4xf32=0
You will get [72 72 72 72]
Run iree-run-mlir b.mlir --iree-hal-target-backends=vulkan-spirv --function-input=1x24x1x4xf32=0 -iree-flow-dispatch-linalg-on-tensors=true -iree-codegen-spirv-experimental-linalg-on-tensors=true
You will get [288 288 288 288].
keep looking...

I probably was wrong on using generic ops that have reduction loop. Let me rethink a bit on it... #4219

The issue is in LinalgFoldUnitExtentDims.
The below snippet is a bit mess, but I couldn't have a better example now.
Input:
func @call(%arg0: tensor<1x24x1x4xf32>, %arg1: tensor<1x24x1x4xf32>) -> tensor<1x4xf32> {
  %cst = constant dense<0.000000e+00> : tensor<f32>
  %0 = linalg.init_tensor [1, 24, 1, 4] : tensor<1x24x1x4xf32>
  %1 = linalg.generic {
    indexing_maps = [affine_map<(d0, d1, d2, d3) -> (d0, d1, d2, d3)>,
                     affine_map<(d0, d1, d2, d3) -> (d0, d1, d2, d3)>,
                     affine_map<(d0, d1, d2, d3) -> (d0, d1, d2, d3)>],
    iterator_types = ["parallel", "parallel", "parallel", "parallel"]}
    ins(%arg0, %arg1 : tensor<1x24x1x4xf32>, tensor<1x24x1x4xf32>)
    outs(%0 : tensor<1x24x1x4xf32>) {
  ^bb0(%arg2: f32, %arg3: f32, %arg4: f32):  // no predecessors
    %5 = addf %arg2, %arg3 : f32
    linalg.yield %5 : f32
  } -> tensor<1x24x1x4xf32>
  %cst_0 = constant 0.000000e+00 : f32
  %2 = linalg.init_tensor [1, 4] : tensor<1x4xf32>
  %3 = linalg.fill(%2, %cst_0) : tensor<1x4xf32>, f32 -> tensor<1x4xf32>
  %4 = linalg.generic {
    indexing_maps = [affine_map<(d0, d1, d2, d3) -> (d0, d2, d3, d1)>,
                     affine_map<(d0, d1, d2, d3) -> (d0, d1)>],
    iterator_types = ["parallel", "parallel", "reduction", "reduction"]}
    ins(%1 : tensor<1x24x1x4xf32>)
    outs(%3 : tensor<1x4xf32>) {
  ^bb0(%arg2: f32, %arg3: f32):  // no predecessors
    %5 = addf %arg2, %arg3 : f32
    linalg.yield %5 : f32
  } -> tensor<1x4xf32>
  return %4 : tensor<1x4xf32>
}

After -linalg-fold-unit-extent-dims
#map0 = affine_map<(d0, d1, d2, d3) -> (d0, d1, d2)>
#map1 = affine_map<(d0, d1, d2, d3) -> (d3)>
#map2 = affine_map<(d0, d1) -> (d0, d1)>
#map3 = affine_map<(d0, d1, d2) -> (d2, d1)>
#map4 = affine_map<(d0, d1, d2) -> (d0)>
module  {
  func @call(%arg0: tensor<1x24x1x4xf32>, %arg1: tensor<1x24x1x4xf32>) -> tensor<1x4xf32> {
    %cst = constant 0.000000e+00 : f32
    %0 = linalg.init_tensor [1, 24, 1, 4] : tensor<1x24x1x4xf32>
    %1 = linalg.tensor_reshape %arg0 [#map0, #map1] : tensor<1x24x1x4xf32> into tensor<24x4xf32>
    %2 = linalg.tensor_reshape %arg1 [#map0, #map1] : tensor<1x24x1x4xf32> into tensor<24x4xf32>
    %3 = linalg.tensor_reshape %0 [#map0, #map1] : tensor<1x24x1x4xf32> into tensor<24x4xf32>
    %4 = linalg.generic {indexing_maps = [#map2, #map2, #map2], iterator_types = ["parallel", "parallel"]} ins(%1, %2 : tensor<24x4xf32
>, tensor<24x4xf32>) outs(%3 : tensor<24x4xf32>) {
    ^bb0(%arg2: f32, %arg3: f32, %arg4: f32):  // no predecessors
      %9 = addf %arg2, %arg3 : f32
      linalg.yield %9 : f32
    } -> tensor<24x4xf32>
    %5 = linalg.init_tensor [4] : tensor<4xf32>
    %6 = linalg.fill(%5, %cst) : tensor<4xf32>, f32 -> tensor<4xf32>
    %7 = linalg.generic {indexing_maps = [#map3, #map4], iterator_types = ["parallel", "reduction", "reduction"]} ins(%4 : tensor<24x4x
f32>) outs(%6 : tensor<4xf32>) {
    ^bb0(%arg2: f32, %arg3: f32):  // no predecessors
      %9 = addf %arg2, %arg3 : f32
      linalg.yield %9 : f32
    } -> tensor<4xf32>
    %8 = linalg.tensor_reshape %7 [#map2] : tensor<4xf32> into tensor<1x4xf32>
    return %8 : tensor<1x4xf32>
  }
}

One reduction loop in the second generic op should be removed because:
In the beginning d0 -> 1, d1 -> 4, d2 -> 24, d3 -> 1, which means reduction loops are d2 and d3. Now there is no unit dim, so there should be one reduction loop.
Actually if you change the input to
func @call(%1: tensor<1x24x1x4xf32>, %arg1: tensor<1x24x1x4xf32>) -> tensor<1x4xf32> {
  %cst_0 = constant 0.000000e+00 : f32
  %2 = linalg.init_tensor [1, 4] : tensor<1x4xf32>
  %3 = linalg.fill(%2, %cst_0) : tensor<1x4xf32>, f32 -> tensor<1x4xf32>
  %4 = linalg.generic {
    indexing_maps = [affine_map<(d0, d1, d2, d3) -> (d0, d2, d3, d1)>,
                     affine_map<(d0, d1, d2, d3) -> (d0, d1)>],
    iterator_types = ["parallel", "parallel", "reduction", "reduction"]}
    ins(%1 : tensor<1x24x1x4xf32>)
    outs(%3 : tensor<1x4xf32>) {
  ^bb0(%arg2: f32, %arg3: f32):  // no predecessors
    %5 = addf %arg2, %arg3 : f32
    linalg.yield %5 : f32
  } -> tensor<1x4xf32>
  return %4 : tensor<1x4xf32>
}

You will get the right IR:
#map0 = affine_map<(d0, d1, d2, d3) -> (d0, d1)>
#map1 = affine_map<(d0, d1, d2, d3) -> (d2)>
#map2 = affine_map<(d0, d1, d2, d3) -> (d3)>
#map3 = affine_map<(d0, d1, d2) -> (d1, d2, d0)>
#map4 = affine_map<(d0, d1, d2) -> (d0)>
#map5 = affine_map<(d0, d1) -> (d0, d1)>
module  {
  func @call(%arg0: tensor<1x24x1x4xf32>, %arg1: tensor<1x24x1x4xf32>) -> tensor<1x4xf32> {
    %cst = constant 0.000000e+00 : f32
    %0 = linalg.tensor_reshape %arg0 [#map0, #map1, #map2] : tensor<1x24x1x4xf32> into tensor<24x1x4xf32>
    %1 = linalg.init_tensor [4] : tensor<4xf32>
    %2 = linalg.fill(%1, %cst) : tensor<4xf32>, f32 -> tensor<4xf32>
    %3 = linalg.generic {indexing_maps = [#map3, #map4], iterator_types = ["parallel", "reduction", "reduction"]} ins(%0 : tensor<24x1x4xf32>) outs(%2 : tensor<4xf32>) {
    ^bb0(%arg2: f32, %arg3: f32):  // no predecessors
      %5 = addf %arg2, %arg3 : f32
      linalg.yield %5 : f32
    } -> tensor<4xf32>
    %4 = linalg.tensor_reshape %3 [#map5] : tensor<4xf32> into tensor<1x4xf32>
    return %4 : tensor<1x4xf32>
  }
}

One of the unit dim is not deleted, so the iterator_types and indexing maps match the behavior.

cc @MaheshRavishankar

I have a quick fix for small test case, let me see if it applies to all cases.

It fixes all the numerical issues under integrations/tensorflow/e2e/
Let me clean it up, add some tests and send out for review...

Awesome! Thanks for looking into this. Havent studied your IR example, but will do so in a bit.

I think there is a subtle issue here (and I see why your fix works). Take this reduction op
  %4 = linalg.generic {
    indexing_maps = [affine_map<(d0, d1, d2, d3) -> (d0, d2, d3, d1)>,
                     affine_map<(d0, d1, d2, d3) -> (d0, d1)>],
    iterator_types = ["parallel", "parallel", "reduction", "reduction"]}
    ins(%1 : tensor<1x24x1x4xf32>)
    outs(%3 : tensor<1x4xf32>) {
  ^bb0(%arg2: f32, %arg3: f32):  // no predecessors
    %5 = addf %arg2, %arg3 : f32
    linalg.yield %5 : f32
  } -> tensor<1x4xf32>

There are two unit-extent dims d0 and d2. It is fine to drop d0, but dropping d2 is legal only if there is another reduction loop after the current reduction loop. I think we just got lucky with our e2e tests.
If you can integrate this into your patch, that would be great. If not, I will fix that upstream after your patch.

No, we are not lucky, it's broken in unidirectional_lstm model in element-wise fusion (and only on dbg mode...) I feel we probably want to revert https://reviews.llvm.org/D99115 from @antiagainst ...

I wonder why only debug mode. Thats weird.
It shouldnt be too hard to fix that though. When you drop unit-dims need to check the iterator type as an extra check. You said you have a fix in progress. If you can adapt it, then thats good. If not I can fix it upstream.
If there are any failing tests now, then we need to see if those can be disabled temporarily.

Oh, did I opened the pandora box? Sorry I wasn't super clear about the full logic there; just throwed in my case as a test to the modified code and it works fine. :D
Reverting https://reviews.llvm.org/D99115 will cause a series of fallouts. It is used by the consumer elementwise op fusion in MobileNet. Better to fix it properly, given that Mahesh said it shouldn't be hard. We need to pay the waiting time of llvm integration anyway.

Verified that this is fixed in https://github.com/llvm/llvm-project/commit/f4eb681dc37ae84e08579bf96cd2a6f58c44f260
It is passing on both CPU and GPU. (seems that there is an issue on SwiftShader, but it could be another issue)


IP_address script is optimized

Description
Please include a summary of the change and which issue is fixed. List any dependencies that are required for this change.
Fixes #263
Replace issue_no with the issue number which is fixed in this PR
Have you read the Contributing Guidelines on Pull Requests?

[ ] Yes
[ ] No

Type of change
Please delete options that are not relevant.

[x] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[x] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update
[ ] Documentation Update

Checklist:

[x] My code follows the style guidelines(Clean Code) of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I've edited the SCRIPTS.md and link to my code.
[ ] I have created a helpful and easy to understand README.md
[ ] My documentation follows Template for README.md
[x] My changes generate no new warnings
[ ] I have added tests/screenshots(if any) that prove my fix is effective or that my feature works.

@avinashkranjan please review


Feature Request: use "display first name" for users

Background:
UofT has a newer feature for changing your display name (for your first name):

https://sgdo.utoronto.ca/resources/resources-for-trans-people-u-of-t/#Setting Display First Name
Without changing your legal name, you may change your display name:

This should be impacting all UofT accounts
Currently, it does change your name on software integrating with Quercus:
-ex: Quercus itself, email, Microsoft Teams, Bb Collab, Crowdmark

Current Behaviour:
MarkUs uses students' legal names

This appears everywhere in relation to a user profile
This means, to submit a project with a group member, you must disclose your legal name

Expected Behaviour:
MarkUs should have no indication of students' legal names amongst other students. For the purpose of students accessing their own profile or students forming groups with each other, they should only be able to see each others' display first name + legal last name.
For the purposes of professors viewing students on MarkUs, it should ideally show only the display first name (but if relevant for marking, it should also show the students' legal first name).

Hi @dakotamcinnis. MarkUs doesn't have a mechanism to automatically get student information from UofT accounts directly. Student information is instead uploaded by a user with an admin account (usually an instructor or system administrator). Whichever name is uploaded by those users is what ends up in a given MarkUs instance.
In order to change the policy regarding the name uploaded to MarkUs you should contact your instructor directly. If you are in the computer science department at UofT's St. George campus you can email teaching computing (see the contact info here: https://support.cs.toronto.edu/poc/index.html) as well.  If you are at either of the other two campuses, I'm not sure who the right person is to contact but your instructor might know if you ask them.
I'm going to close this issue because the solution is external to MarkUs itself.

@dakotamcinnis I realized that there is a potential solution that can be handled within the scope of MarkUs. Please see #4890 and feel free to add any suggestions/comments there if you would like.


freegeoip's API will be deprecated July 1st, 2018

Announced at https://github.com/apilayer/freegeoip/
freegeoip - Important Announcement
[The old freegeoip API is now deprecated and will be discontinued on July 1st, 2018]
Launched more than 6 years ago, the freegeoip.net API has grown into one of the biggest and most widely used APIs for IP to location services worldwide. The API is used by thousands of developers, SMBs and large corporations around the globe and is currently handling more than 2 billion requests per day. After years of operation and the API remaining almost unchanged, today we announce the complete re-launch of freegeoip into a faster, more advanced and more scalable API service called ipstack (https://ipstack.com). All users that wish to continue using our IP to location service will be required to sign up to obtain a free API access key and perform a few simple changes to their integration. While the new API offers the ability to return data in the same structure as the old freegeoip API, the new API structure offers various options of delivering much more advanced data for IP Addresses.
Required Changes to Legacy Integrations (freegeoip.net/json/xml)
As of March 31 2018 the old freegeoip API is deprecated and a completely re-designed API is now accessible at http://api.ipstack.com. While the new API offers the same capabilities as the old one and also has the option of returning data in the legacy format, the API URL has now changed and all users are required to sign up for a free API Access Key to use the service.

Get a free ipstack Account and Access Key

Head over to https://ipstack.com and follow the instructions to create your account and obtain your access token. If you only need basic IP to Geolocation data and do not require more than 10,000 requests per month, you can use the free account. If you'd like more advanced features or more requests than included in the free account you will need to choose one of the paid options. You can find an overview of all available plans at https://ipstack.com/product

Integrate the new API URL

The new API comes with a completely new endpoint (api.ipstack.com) and requires you to append your API Access Key to the URL as a GET parameter. For complete integration instructions, please head over to the API Documentation at https://ipstack.com/documentation. While the new API offers a completely reworked response structure with many additional data points, we also offer the option to receive results in the old freegeoip.net format in JSON or XML.
To receive your API results in the old freegeoip format, please simply append &legacy=1 to the new API URL.
JSON Example: http://api.ipstack.com/<IP_ADDRESS>?access_key=YOUR_ACCESS_KEY&output=json&legacy=1
XML Example: http://api.ipstack.com/<IP_ADDRESS>?access_key=YOUR_ACCESS_KEY&output=xml&legacy=1
New features with ipstack
While the new ipstack service now runs on a commercial/freemium model, we have worked hard at building a faster, more scalable, and more advanced IP to location API product. You can read more about all the new features by navigating to https://ipstack.com, but here's a list of the most important changes and additions:

We're still free for basic usage

While we now offer paid / premium options for our more advanced users, our core product and IP to Country/Region/City product is still completely free of charge for up to 10,000 requests per month. If you need more advanced data or more requests, you can choose one of the paid plans listed at https://ipstack.com/product

Batch Requests

Need to validate more than 1 IP Address in a single API Call? Our new Bulk Lookup Feature (available on our paid plans) allows you to geolocate up to 50 IP Addresses in a single API Call.

Much more Data

While the old freegeoip API was limited to provide only the most basic IP to location data, our new API provides more than 20 additional data points including Language, Time Zone, Current Time, Currencies, Connection & ASN Information, and much more. To learn more about all the data points available, please head over to the ipstack website.

Security & Fraud Prevention Tools

Do you want to prevent fraudulent traffic from arriving at your website or from abusing your service? Easily spot malicious / proxy / VPN traffic by using our new Security Module, which outputs a lot of valuable security information about an IP Address.
Next Steps

Deprecation of the old API

While we want to keep the disruption to our current users as minimal as possible, we are planning to shut the old API down on July 1st, 2018. This should give all users enough time to adapt to changes, and should we still see high volumes of traffic going to the old API by that date, we may decide to extend it further. In any case, we highly recommend you switch to the new API as soon as possible. We will keep you posted here about any changes to the planned shutdown date.

Any Questions? Please get in touch!

It's very important to ensure a smooth transition to ipstack for all freegeoip API users. If you are a developer that has published a plugin/addon that includes the legacy API, we recommend you get in touch with us and also share this announcement with your users. If you have any questions about the transition or the new API, please get in touch with us at<EMAIL_ADDRESS>

FYI- ipstack's free option doesn't support HTTPS

Dealt with here


Big integer issue

It seems that elasticdump cut of big integer value.
If I have in source field
{
 "content_id":<PHONE_NUMBER>4031742**66**,
}

after copy
I get
{
  "content_id":<PHONE_NUMBER>9858125**00**
}

I guess this is JavaScript int/long restriction.

 My steps:

I tried with curl it returned<PHONE_NUMBER>4031742**66**
Then tried to export in json file and got<PHONE_NUMBER>4031742**00**

I'm not proficient with JS but I noticed that another nodejs software has the same issue (Insomnia, ES web plugin).
I'll try to prepare usefull dataset later.

So are there any plans to fix it?
Here
https://stackoverflow.com/questions/18755125/node-js-is-there-any-proper-way-to-parse-json-with-large-numbers-long-bigint
there is advice to use 3-d party json parse libs.

You can try using https://www.npmjs.com/package/json-bigint. It will patch JSON.parse to allow big int parsing...

Someone already submitted an old PR for this. I will have a look, add a few tests and push something for you to test. https://github.com/taskrabbit/elasticsearch-dump/pull/360

It would be great!

@evantahle that's a better field name.

But why we can not use any big int library (BigInteger.js ?) to store value instead strings? If you guess that it could be slower then probably it should be turned on by additional key like --support-big-int.
I'm not sure that every user knows list of fields with big int, and wants to explore data source to find it, but could agree for some extra time delay for properly working.

Yea, but it's very aggressive. It would convert all numeric fields to big integer. Event an ID field. e.g. "121212". If you have a string mapping then this would fail, which isn't great.

That's why I suggested additional key :-)


Update local build process to use Gradle scripts

Right now it's done with a bunch of Shell scripts, which can probably be reused, but I want to capture everything in Gradle cause why not? We can have the Gradle scripts build the processes, and test them prior to doing the local deployment, just to ensure functionality

Most of the module specific gradle tasks exist now. What's left is the CDK (not required at all), and translating the existing root-level scripts for running the project locally


chore: add changeset for pr #1234

What/Why/How?
Add changeset for pr #1234
Reference
https://github.com/Redocly/redocly-cli/pull/1234
Testing
Screenshots (optional)
Check yourself

[ ] Code is linted
[ ] Tested with redoc/reference-docs/workflows (internal)
[ ] All new/updated code is covered with tests

Security

[x] Security impact of change has been considered
[ ] Code follows company security practices and guidelines

Do the changes impact openapi-core?


LORE ONNX Export

Does anyone know how to export LORE ONNX? I applied the same export with CenterNet but without success.

You can try to use the LORE ONNX model in the repo TableStructureRec

@SWHL Hi, It is great work! However, I just want to export Lore to ONNX and I want to use my own Table Recovery due to low performance when using with large and complex table. Can you share how to convert to ONNX for me ?

The LORE onnx model was converted from my previous job, and the conversion envrionment was lost at that time.
This weekend, I wiil try to see if I can convert successfully again.

@Holmes2002 hello, I reorganized the conversion code, and there is no problem in my test here. You can have a try.
This is the repo link: https://github.com/SWHL/ConvertLOREToONNX

Hi, Thank you for your repo to convert Lore to ONNX. However, seem that you converted model ckpt_wireless-Resnet 18 to ONNX. I want to convert models DLA-34 to ONNX. Have you tried converting the DLA model to ONNX?

I haven't tried DLA before.

nice

@Holmes2002
Have you had any information for DLA-34(onnx)?

@phamkhactu No, I successfully converted DLA-34 to ONNX but the inference doesn't work and I don't understand why

@phamkhactu No, I successfully converted DLA-34 to ONNX but the inference doesn't work and I don't understand why

Could you give me snip code for converting onnx if convenience? Or you can compare your onnx with onnx themself. I don't know it extractly which is  by a author.

@phamkhactu Hi, this is code for converting to ONNX in final file
https://github.com/Holmes2002/Table-Recognition/blob/main/LORE-TSR/src/lib/models/networks/pose_dla_dcn.py

My own inference code but did't work
https://github.com/Holmes2002/Table-Recognition/blob/main/LORE-TSR/src/lib/models/networks/inference_onnx.py

You can check and modify code. Announce to me if you success


Should be able to pass CLI args that will supersede those in a saved Trainer config

The current behavior of the ML CLI is, if a Trainer config JSON file is passed, we skip processing all other CLI args unless we also want to overwrite the config file, in which case we don't even look at that file. I think the desired behavior is instead to load from the config file if provided, and then any additional args that were passed via CLI will supersede anything in the config file. Overwriting the existing config file will still remain optional.

this was fixed by #933


Load Model Config and Column Config Once in Multi Threaded Mapper Task

https://github.com/ShifuML/shifu/blob/master/src/main/java/ml/shifu/shifu/core/varselect/VarSelectMapper.java#L163
https://github.com/ShifuML/shifu/blob/master/src/main/java/ml/shifu/shifu/core/varselect/VarSelectMapper.java#L165
If over 500 mappers to read the same HDFS ColumnConfig.json and model files, the read is pretty slow in some tasks. Try to make in each mapper, such reading only once as static instance.
CorrelationMapper is the same.

Leverage local cache is also a good solution

Done in correlation and varselect sensitivity mappers


update Userguide, Showdue, answer, schedule

Pull Request Test Coverage Report for Build 363

0 of 0 (NaN%) changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 87.928%

Totals

Change from base Build 360:
0.0%

Covered Lines:
2185

Relevant Lines:
2485

💛  - Coveralls


unable to quit q, control-c, esc does not work on mac with brew

Please follow the guide below

You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your issue (like this: [x])
Use the Preview tab to see what your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.08.04. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

[x] I've verified and I assure that I'm running youtube-dl 2018.08.04

Before submitting an issue make sure you have:

[x] At least skimmed through the README, most notably the FAQ and BUGS sections
[x] Searched the bugtracker for similar issues including closed ones
[ x] Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

[x] Bug report (encountered problems with youtube-dl)
[ ] Site support request (request for adding support for a new site)
[ ] Feature request (request for a new functionality)
[ ] Question
[ ] Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:
Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):
i dont have a log since this bug does not produce an error

the video i was trying is https://www.youtube.com/watch?v=MZ_s8zpLYRI&t=7052s

i am on mac osx 10.12 using homebrew and while i can download live streams ok i am unable to stop the stream

i tried the q command and nothing it still continues to download i even tried holding q incase the cpu demand is so high that the keyboard is not being registered and eventually the command will go through   still not happening.

so i try control-c to quit the process like you do on linux and still nothing.

i tried the esc key and nothing.

if i quit or close the terminal i am able to stop the stream but it leaves the files unclosed properly so there are 2 files a  part and a ytd file.

however if i try a normal live stream like https://www.youtube.com/watch?v=ytJb-xDl69U

the q does not work but holding control-c does stop and close the file properly giving me a playable file

the first video is 300+ hours long and i wanted to download a few minutes so i could calculate how big it is but i suspect it may be using a different streaming method than most others

if i try a normal youtube video and then control-c while it does not close the file properly and leaves a part file however quicktime is able to play the partial file if i rename it to omit the "part" from the name

every thing is up to date as i am able to get the stream capture to work without problems
thanks for the help

If the purpose of this issue is a site support request please provide all kinds of example URLs support for which should be included (replace following example URLs by yours):

Single video: https://www.youtube.com/watch?v=BaW_jenozKc
Single video: https://youtu.be/BaW_jenozKc
Playlist: https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc

Note that youtube-dl does not support sites dedicated to copyright infringement. In order for site support request to be accepted all provided example URLs should not violate any copyrights.

Description of your issue, suggested solution and other information
Explanation of your issue in arbitrary form goes here. Please make sure the description is worded well enough to be understood. Provide as much context and examples as possible.
If work on your issue requires account credentials please provide them or explain how one can obtain them.

Ctrl+C should work fine. It may take some time though.

nope that did not work it continued to download until  it finished normally.
it looks like it is not as live as it claims
\
On Aug 11, 2018, at 4:27 PM, Sergey M. wrote:

Ctrl+C should work fine. It may take some time though.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.


<IP_ADDRESS>.0

This PR updates the partner version to <IP_ADDRESS>.0 for release.
DO NOT MERGE

Closing this PR to make it fresh directly off of <IP_ADDRESS>.1 instead.

Ok, I cannot since there are branch protections.

Released: https://github.com/ChartBoost/chartboost-mediation-android-adapter-chartboost/releases/tag/v<IP_ADDRESS>.0


layer type for smart object layers is not correct

In both doc info and in change events, the layer "type" for smart object layers (both embedded and linked) is reported as "layer".
There's evidence in our code that at one point, the reported type was "smartObjectLayer": https://github.com/adobe-photoshop/generator-assets/blob/b76afe383d61d7c78d6764ddd9df9b90031bc61b/lib/dom/layer.js#L1234
Regardless of what may have changed, we are now no longer constructing the right DOM object types for smart object layers. A workaround on the JS side would be to look for the smartObject property of a layer as a way to set the type. (This is what the unmerged PR #298 did.)

Assigning to myself to check this out.

To my eye, it looks like we are already looking at the smartObjects property and handling appropriately.  One question: In the unused code path (if layer.type were to equal smartObjectLayer) it seems to include _setTimeContent see here.  @iwehrman is this ringing any bells?
Demoting this to "code cleanup" to remove the unnecessary code.

I don't have a clear recollection, but I remember that there were some parts of the DOM that I started building but which didn't have much chance to test or use.


Enable adding data disks

From https://github.com/MSOpenTech/azure-sdk-for-go/blob/master/management/vmutils/datadisks.go#L11 there's a function to add a data disk to a VM. Could this functionality be added to packer-azure?

Sounds like a good feature. We'll need #57 to merge first...

This would be very helpful.  I need the ability to add extra disks to my VMs.  This is kind of a show stopper.

@ghost @camechis : do you need 'new'/'empty' data disks, already existing vhd's, or a mix?

@paulmey I am thinking new/empty.  My use case is I need to provision the vms that can be used for some type of NoSQL store.  With the default vms I need to mount a disk ( eg. /var ) that can have large amounts of storage.

See if #91 gets the show going for y'all. I can see if I can maybe add an option to specify things like host caching type if you need that?

#91 I believe gets us there.


refactor(parser): macro for ASCII identifier byte handlers

Add a macro for ASCII identifier byte handlers.
This is a preparatory step towards #2352.

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

Current dependencies on/for this PR:

main

PR #2350

PR #2351   👈

PR #2352

This stack of pull requests is managed by Graphite.

Merge activity

Feb 8, 10:50 PM EST: @Boshen started a stack merge that includes this pull request via Graphite.


Incomprehensible sql3 tracks

Hi !
Recently I am build FluxEngine by description on cowlark.com website.
I need to read floppy disks in a format that is NOT supported by FluxEngine.
I am using the command:
fluxengine.exe read ibm --write-flux xxx.sql3
then I use my parser on the xxx.sql3 file (ibm.img has 0 byte size).
Some tracks are decoded correctly, but they are not on their numbers: the cylinder number in the SQL does not correspond info into address fields (I have not review this problem yet).
But some tracks in the SQL are filled only with the constant 0x3F (and it compressed size 69-70 bytes). But it not be possible: is not a single impulse on the track? And no INDEX ?
Is there any reason why such a file might be generated?
And this happens only at where side=0, for side=1 no such failures.
Floppy drive is ok.
I can send sql3-files for research.
Thanks.

Re mismatched address info: that's normal for some formats; the cylinder number in the SQL file corresponds to the physical location of the head, which doesn't necessarily correspond to the logical cylinder encoded in the address fields. You'll see this in, for example, 40-track disks.
The blank tracks are odd, though. It suggests that the data isn't being transferred correctly from the drive to the board. When you do the read (and remember to do --retries=0), does the total amount of data look reasonable --- 200 to 300ms? Also, do you have more than one drive plugged in? I've seen termination oddities in that situation.

Thanks for your reply !

--retries=0
I have not used this key, can this make the oddities?

Empty tracks have a slightly different size around 47100-47300 bytes. Since 0x3F is 83 ns, then 63 * 0.083 * 47200 = ~ 246809 μs.
I'm store two disk images here:
https://drive.google.com/drive/folders/1P8sODlU74WES_PzsxB3P5poDBV-dTksi?usp=sharing
The error with track numbering is more complex.
For example, the d2.sql3 image:
t0h0, t0h1 are empty.
t1h0, t1h1 - correctly decoded, address field CRC ok, data field CRC ok, but this data should be located on t5h0 and t5h1.
t2h0, t2h1 - ok, but should be located on t4h0 and t4h1.
t3h0, t3h1 - all right!
t4h0 - empty
t4h1 is ok, but should be located on t2h1.
...etc
I don't understand how such a strange address glitch occurs. Please note: the head numbers is always correct. But the tracks are numbered in reverse.
I am well knowing of this format, I can also read (and write) these discs on another device and I know that these discs are "standard" - they do not have any deviations in the recording rules.
I use the following script to retrieve data from SQL:
`#!/bin/sh
for img in *.sql3
do
dir="${img%%.sql3}"
[ -d "$dir" ] && continue
mkdir "$dir"
for t in seq -f "%02g" 0 79
do
for s in seq 0 1
do
printf "select writefile('/tmp/zdata.zl', data) from zdata where track=$t and side=$s;\n" | sqlite3 $img
trk="${dir}/t${t}s${s}.trk"
  # This command decompress ok
  ./zlib-decompress.py /tmp/zdata.zl "$trk" || rm "$trk"

  # This command will output "CRC ERROR", but decompress be right:
  # cat GZIP-HEADER8 /tmp/zdata.zl | gunzip > "$trk"

  [ ` stat -f %z /tmp/zdata.zl ` -lt 100 ] && cp /tmp/zdata.zl ${trk}.z
  rm /tmp/zdata.zl
done

done
done
`
Little program zlib-decompress.py:
`
#!/usr/local/bin/python2.7
import zlib
import sys
with open(sys.argv[1], 'r') as f:
z = f.read()
s = zlib.decompress(z)
with open(sys.argv[2], 'w') as f:
f.write(s)
`
And then I use the pascal-program to decode the files t??h?.trk from MFM bitstream to a RAW-byte stream.
(RAW-byte stream: stream between real hardware floppy disk controller and software driver. It's internal format for our disk images).

A solved it !! YES !
At short: power supply problem.
Some floppy disks rotate easily in an envelope, a floppy drive (Samsung SFD-321B / KE) consumes about 250 ma. However, some floppy disks rotate harder and make noise. The drive consumes about 350 ma.
When the head move to next cylinder, the voltage drops and some kind of failure occurs: FluxEngine starts to move the head back (I don't know why, FluxEngine is powered independently, from USB. The track counter increases on the screen). When the head returns to "0", FluxEngine moves it to the desired track and then the head moves to the center again, counter on screen again in sync with head position.
0x3F-fill track zero tracks are probably due to the fact that due to a power failure:
the floppy drive closes the read channel (wait for stabilize RPM), so the FluxEngine does not actually receive pulses.
15 test floppies read successfully.
Great device !

Ah, yes. The heads will be moving backwards because the low voltage will be causing missteps --- it's a problem with stepper motors. Then the head won't be pointing at any track and you'll get bad data. You might try 'fluxengine test voltages' to read the various signal levels.
If you're sorted, I'll close this issue --- feel free to reopen if necessary.


Fix typo in readme

Codecov Report

Merging #604 (51c4f07) into main (10ef28d) will not change coverage.
The diff coverage is n/a.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@           Coverage Diff           @@
##             main     #604   +/-   ##
=======================================
  Coverage   99.09%   99.09%
=======================================
  Files          12       12
  Lines         442      442
  Branches       91       91
=======================================
  Hits          438      438
  Misses          4        4

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


No corr_inv_la.csv

Hello,
When I try to train or test, I get the error that data/corr_inv_la does not exist. Are there some extra files we need to download or preprocess to run the script?

Hello,
Thanks to report the issue! Could you check again by pulling the master branch?

It is working now. Thank you!
On Tue, Feb 28, 2023 at 1:46 PM Semin Kwak @.***> wrote:

Hello,
Thanks to report the issue! Could you check again by pulling the master
branch?
—
Reply to this email directly, view it on GitHub
https://github.com/semink/TwoResNet/issues/7#issuecomment-1448890650,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AG2HMHSJQTWUAF4VNMH6Z63WZZPYZANCNFSM6AAAAAAVLFGBFY
.
You are receiving this because you authored the thread.Message ID:
@.***>