Character Creation Bug

BUG DESCRIPTION
A deleted and then remade character with the same name does not start out as a new character and instead retains some information/settings of the previously deleted character.
To Reproduce

Create a character named "Shazam"
Enter the world. (Outbreak)
Use /mapmenu to transition to Boomtown (or any other map)
Delete that character.
Make a new character, also named "Shazam"
Enter the world.

Expected behavior
The character expects to enter the Outbreak map.
Actual behavior
The character unexpectedly starts in Boomtown. It is possible to repeat this behavior of deleting and remaking a character to log into the previous character's last map.
Note that deleting the character, then shutting down and restarting the server before remaking him, results in the expected behavior. (i.e., The character spawns in Outbreak.)
SEGS Version
0.6.1

Thank you for reporting!

Can you confirm that this still happens if you leave the server running but completely restart the client?

I've tested two ways with varied results. The first deletes the character before restarting the client, the second after restarting.
First Method

Create a character named "Blue Brute".
Enter the world. (Outbreak)
Use /mapmenu to travel to Galaxy City.
Delete the Character.
Exit and then restart the client.
Make a new character named "Blue Brute".
Enter the world.

Result
In this case, it behaves properly. The newly remade "Blue Brute" starts inside Outbreak.
Method Two

Create a character named "Blue Brute".
Enter the world. (Outbreak)
Use /mapmenu to travel to Galaxy City.
Exit and then restart the client.
Delete the "Blue Brute".
Make a new character named "Blue Brute".
Enter the world.

Result
In this case, the bugged behavior appears and the character starts in Galaxy City instead of Outbreak.

Method 1and 2 appear to be the same? Typo?
My fear is that this is a client bug particular to our version. It warrants further investigation, but it seems that something is stuck in memory and the client recognizes the character name and makes assumptions about the character (loading prior zone, GUI, etc.)

The difference is deleting the character before or after restarting the client.

Does the bug appear only when creating the new character in the same slot? Or does it work normally in a different slot?

That hints at this being server side, with our login system failing to reset memory associated with a particular slot at charsel.
I'll have to dig in further, but feel free to propose a solution if you (or anyone else) finds one.
Appreciate you filing this report @YellowAdder
and I greatly appreciate @0Anarchium0 and @Magentah for assisting with testing and reviewing this as well.

Same character name is a red herring, it seems to be the slot idx , still investigating, but if you delete character "Bob" from slot 0, then create a new character called "Miles" in slot 0, the same issue occurs regarding map

@nemerle @broxen
MapInstance is not being reaped, the map name still shows in the client title bar "City Of Heroes: Talos Island" once back at login screen... could this play any part?

Looks like the map session is never marked for reaping in MapInstance::on_disconnect .. am I on the right tracks here?
From reviewing output, map is spun up on first connect with fresh launch of client, after log out (no other entity on map), then log in again with no exit or restart, map isn't spun up again so I'm assuming it's never spun down? My understanding is it should be if no other players on the map?

MapInstance::spin_down() is commented out and not implemented currently. I imagine this is intended to handle the session reaping.

Thanks Magentah. Shouldnt reaping happen on disconnect rather than spin down as we are only reaping the client that's disconnecting at that point?

Well when a user disconnects it should reap their session, but perhaps during a character switch it's keeping that session alive and assuming the same values for entitydata.
spin_down would be when the MapInstance is destroyed, which I thought happened when everyone left the instance, but the reality is that it should probably happen only if the instance is empty for longer than X amount of time. Maybe a few hours, or maybe customizable.
I would guess that the issue here is in reaping the link when a user disconnects.

I think that reaping is automatic based on a timed interval, setup during MapInstance::start(). This automatically reaps all the stored sessions that are inactive.
The bug may be logging back in before the session is reaped?

I could be misunderstanding the code, but it looks like we have to mark_for_reaping before it gets picked up by the auto reaping methods. For instance similar code in GameInstance, on disconnect mark_for_reaping is used but it isnt done currently in MapInstance on disconnect

Ah quite possibly, I’m not at home so looking through on mobile browser, I could be reading it wrong. I can take a look at fixing this soon if no one else takes it.

I'll give it a bash first if that's okay? Want to get stuck into the game code a bit more so will probably be a good learning experience for me 😁

Sure! I may be able to answer questions, I done some stuff with sessions before for some transfers.

Okay nice thank you 😁


Documentation for R2R/RxR Vision-and-Language Navigation

❓ Questions and Help
Hey folks!
I've been following the work on Habitat for a while, and think it is an incredible platform. I'm spinning up a few research projects here at Stanford that require using the Room2Room (and offshoots like R4R, RxR) datasets, and the graph-based navigation (used in papers like this).
I'm wondering if there's any documentation/tutorials for getting started with R2R in Habitat-Lab, and additionally, if you could point me to any VLN baselines that exist built on top of the Habitat ecosystem.
Thanks so much!

Hey @siddk, long time to no see. :)
There are some baselines/examples that use R2R for VLN that have tests that can be found here. It's pretty easy to add dataloaders for other variants like R2R.
Off the top of my head, @jacobkrantz has a nice VLN code baseline here.

Hey @siddk, long time to no see. :)
There are some baselines/examples that use R2R for VLN that have tests that can be found here. It's pretty easy to add dataloaders for other variants like R2R.
Off the top of my head, @jacobkrantz has a nice VLN code baseline here.

Haha hey @Skylion007 - it's been a while!
Thanks for the pointer, I'll check out the example.
The VLN-CE codebase by @jacobkrantz is for continuous environments - rather than navigating a set of discrete nodes in a MP3D environment, the space is continuous. For our work (comparing against existing baselines), we need the discrete environment; is this supported in Habitat-Lab (it isn't clear from this example)?
Thanks!

Haha hey @Skylion007 - it's been a while!
Thanks for the pointer, I'll check out the example.
The VLN-CE codebase by @jacobkrantz is for continuous environments - rather than navigating a set of discrete nodes in a MP3D environment, the space is continuous. For our work (comparing against existing baselines), we need the discrete environment; is this supported in Habitat-Lab (it isn't clear from this example)?
Thanks!

Yeah, actually discrete environments are the default. Continuous environments are relatively new (and somewhat experimental API). Should still be a lot of code in that repo that is relevant though.
Here is the config for the discrete VLN task baseline: https://github.com/facebookresearch/habitat-lab/blob/7766bf5fdb70db2aede35d2eb1123662a485f7f8/configs/tasks/vln_r2r.yaml

Yeah, actually discrete environments are the default. Continuous environments are relatively new (and somewhat experimental API). Should still be a lot of code in that repo that is relevant though.
Here is the config for the discrete VLN task baseline: https://github.com/facebookresearch/habitat-lab/blob/7766bf5fdb70db2aede35d2eb1123662a485f7f8/configs/tasks/vln_r2r.yaml

Navigating to various discrete nodes can be easily done be passing the proper action_args to the TeleportAction define in the repo.

Navigating to various discrete nodes can be easily done be passing the proper action_args to the TeleportAction define in the repo.

discrete environments are the default.

Actually continuous environments are the default in VLN-CE. @Skylion007 I think you are confusing discrete space with discrete actions. The VLN-CE task is continuous space but discrete actions.
@siddk I don't know of anyone doing straight VLN via node-to-node navigation in Habitat yet, but it is possible to implement via TeleportAction. You would also need to port the connectivity graph of R2R to Habitat which could be accomplished by mapping the location coordinates of each node from Matterport3D-Sim to Habitat. Not all of those locations are immediately navigable by a ground agent in continuous space, but depending on your use case you may or may not care. I'm happy to follow up further on this.

discrete environments are the default.

Actually continuous environments are the default in VLN-CE. @Skylion007 I think you are confusing discrete space with discrete actions. The VLN-CE task is continuous space but discrete actions.
@siddk I don't know of anyone doing straight VLN via node-to-node navigation in Habitat yet, but it is possible to implement via TeleportAction. You would also need to port the connectivity graph of R2R to Habitat which could be accomplished by mapping the location coordinates of each node from Matterport3D-Sim to Habitat. Not all of those locations are immediately navigable by a ground agent in continuous space, but depending on your use case you may or may not care. I'm happy to follow up further on this.

Ahh gotcha - thanks for saving me @jacobkrantz!
Node-to-Node navigation is ideally what I'd want to compare against some specific baselines (although we're definitely planning on moving to more of a VLN-CE setup at some point!).
Am I correct in assuming the best way to do this at the moment is through Peter Anderson's original Simulator (via the Docker instructions here)? Or is there something better?
It's interesting that this hasn't been folded into Habitat-Lab yet; I feel like a lot of recent work is still working on node-to-node navigation (R4R, RxR and affiliated models).

Ahh gotcha - thanks for saving me @jacobkrantz!
Node-to-Node navigation is ideally what I'd want to compare against some specific baselines (although we're definitely planning on moving to more of a VLN-CE setup at some point!).
Am I correct in assuming the best way to do this at the moment is through Peter Anderson's original Simulator (via the Docker instructions here)? Or is there something better?
It's interesting that this hasn't been folded into Habitat-Lab yet; I feel like a lot of recent work is still working on node-to-node navigation (R4R, RxR and affiliated models).

If you want to make direct comparisons against specific VLN baselines, then sticking with the original VLN R2R task would be an easier place to start. Most works have used https://github.com/peteanderson80/Matterport3DSimulator, but you could also look into https://github.com/google-research/valan. If you wanted to do this in Habitat, recreating VLN would deviate from the VLN R2R task. Chiefly, there is a discrepancy of using 3D scene reconstructions (Habitat) vs using a graph topology of Matterport camera scans (VLN R2R task definition). Even if you match camera settings, there is some loss in visual fidelity from using reconstructions. If you then wanted direct comparison to VLN baselines, you would need to re-run them in your new Habitat setting.
This is just guiding you in the direction of least resistance -- I think it would be interesting to see node-to-node VLN in Habitat!

If you want to make direct comparisons against specific VLN baselines, then sticking with the original VLN R2R task would be an easier place to start. Most works have used https://github.com/peteanderson80/Matterport3DSimulator, but you could also look into https://github.com/google-research/valan. If you wanted to do this in Habitat, recreating VLN would deviate from the VLN R2R task. Chiefly, there is a discrepancy of using 3D scene reconstructions (Habitat) vs using a graph topology of Matterport camera scans (VLN R2R task definition). Even if you match camera settings, there is some loss in visual fidelity from using reconstructions. If you then wanted direct comparison to VLN baselines, you would need to re-run them in your new Habitat setting.
This is just guiding you in the direction of least resistance -- I think it would be interesting to see node-to-node VLN in Habitat!

Thanks so much @jacobkrantz! Really appreciate it. In the meantime, I'll proceed with the existing Simulator, and maybe re-implement the VALAN wrapper.

Thanks so much @jacobkrantz! Really appreciate it. In the meantime, I'll proceed with the existing Simulator, and maybe re-implement the VALAN wrapper.

@siddk If you do reimplement the wrapper, feel free to submit a PR.

@siddk If you do reimplement the wrapper, feel free to submit a PR.


Bootstrapping connections, info exchange and multicast

Hey there,
Trying to get nodes communicating and ran into a few issues after trying to get somewhere past the examples.

Without copy & pasting public connection information, how do two nodes prepare the connection in order to connect? I've tried adding bootstrap contacts which when doing bootstrap_accept() / bootstrap() get a 'peer' structure in the Future - splitting this into sink / stream and doing send or receive from either end doesn't seem to work.

What is the expected way to do multicast? I can't seem to find any code which does this in the repo but maybe I'm blind.

Fixed the first issue doing this on the bootstrap node:
        println!("Spawning bootstrap acceptor .");
        handle.spawn(
            service
                .bootstrap_acceptor()
                .for_each(|peer| {
                    let (peer_sink, peer_stream) = peer.split();
                    peer_stream
                        .for_each(|data: Vec<u8>| {
                            println!("Received: {}", unwrap!(String::from_utf8(data)));
                            Ok(())
                        })
                        .then(|_| Ok(()))
                    // Ok(())
                })
                .then(|_| {
                    Ok(())
                }),
        );

And this on a regular node:
        let bootstrap_peer: Peer<NodeId> = unwrap!(
            event_loop.run(
                service.bootstrap(
                    HashSet::new(),
                    service_discovery,
                    CrustUser::Client,
                )
            )
        );

        let (peer_sink, peer_stream) = bootstrap_peer.split();

        println!("Sending hello ...");
        unwrap!(
            event_loop.run(
                peer_sink
                    .send((1, format!("hello").into_bytes()))
                    .and_then(|peer_sink| {
                        println!("Sending ok ...");
                        peer_sink.send((1, format!("ok").into_bytes()))
                    }),
            )
        );

So it seems as if bootstrap connections are already connected and you can just send / receive.

Hey, Edward, indeed, when we're bootstrapping from some node, Crust uses hard coded contacts. These contacts have d is their public address and public key which is all we need to securely connect to some peer. So no connection info is required and it is not exchanged later. But note that we can bootstrap only from publicly accessible nodes - no hole-punching is executed in this case.
Multicast is out of scope of Crust. Crust has no sense of network, it only deals with two peer communication.
Could you tell a bit about your usecase?

We have peer-discovery over LAN which uses broadcast (ServiceDiscovery) anyway

Hey, Edward, indeed, when we're bootstrapping from some node, Crust uses hard coded contacts. These contacts have d is their public address and public key which is all we need to securely connect to some peer. So no connection info is required and it is not exchanged later. But note that we can bootstrap only from publicly accessible nodes - no hole-punching is executed in this case.

Thats good information, thanks.

Multicast is out of scope of Crust. Crust has no sense of network, it only deals with two peer communication. Could you tell a bit about your usecase?

Sure, I'm basically experimenting with a leaderless consensus model. Some of the operations in the model require multicast in the sense of having an operation that can just send data across all active connections. I'm using a basic kademlia router (custom) at the moment for node discovery and acquiring a set of active connections. From there theres the naive way of just looping across the connections and sending some data but I'm trying to get away from that - ideally a multicast primitive in Crust could have been used which sends data to N active connections simultaneously. Barring that I might try something like mpsc crossbeam or investigate further down that path.

yeah, Crust doesn't keep a list of alive connections. You have to manage them yourself. But probably it shouldn't be too hard to implement a multicast function :)


does the classification process not use softmax?

I re-read the Forward function for sequence classification and it turns out there is no external application of softmax. Do you really not use softmax, or do you use softmax internally? If you use softmax internally, where is the softmax implemented?

@SamuelCahyawijaya

@febrianti2602 : Why do we need softmax? If we want to get the probability distribution, we can apply softmax to the logits, but in our case:

During inference, since we just want to get the most probable class, so we can simply get top-1 of the logits (as softmax is a monotonic function)

During the training, we use the CrossEntropyLoss which -- as specified in the documentation itself -- accepts logits as its input and performs a log softmax followed by negative log-likelihood to the input logits.

Hope it helps.


gmsync insists on syncing a song despite it being identical

First of all, thanks for those scripts - as much as I enjoy googlemusic, the default down/up/sync experience is pretty horrible.
I've ran a test gmsync -m -d up from my home collection to GM, and I've discovered that gmusicapi would like to upload some files despite both side having (seemingly) the same information. Specifically, one of the outstanding albums was an album that I've bought recently on GM, then downloaded to local collection. Is there a way to get some information why gmsync decided to declare a mismatch? Would a difference in tags be sufficient (even with -m)? Adding -l doesn't really give me any more information.

First of all, thanks for those scripts - as much as I enjoy googlemusic, the default down/up/sync experience is pretty horrible.

Way to provide a ton of feedback on that one. Most people find the default experience pretty good.


Tensorflow implementation

Is there any tensorflow implementation of the code ?

We are training TF versions now. Should be ready to commit in a few weeks.

can you please share the dataset for age?

We use these sets
https://data.vision.ee.ethz.ch/cvl/rrothe/imdb-wiki/
http://gesture.chalearn.org/2016-looking-at-people-cvpr-challenge

are you using imdb celebrity faces? because the cropped face which of 7 gb is not proper. the age of people is not exact. I will share with you the code that I used to segregate the images into various folders based on their age. after segregation, I noticed that even the teenagers are in old people age folder. However imdb wiki dataset it good enough..
import numpy as np
from utils import get_meta
import cv2
import os
from shutil import copy
db = "imdb"
mat_path = "data/{}_crop/{}.mat".format(db, db)
print (mat_path)
full_path, dob, gender, photo_taken, face_score, second_face_score, age\
    = get_meta(mat_path, db)

print("#images: {}".format(len(face_score)))
print("#images with inf scores: {}".format(np.isinf(face_score).sum()))

img_paths = []
photo_root = "data/{}_crop/".format(db)

for i in range(len(age)):
    if age[i] == 7:
        print (photo_root +str (full_path[i][0]))
        img_cv = cv2.imread(os.path.join(photo_root, full_path[i][0]))

        copy(photo_root + "/" + str(full_path[i][0]), "seven")

@mahehu any ideas as of how I can get this done?

Can please share with us more details (files to compile ...) ?

I think all these issues are addressed in the current version.


flake in GetAndCloseStreamWithTracing

test touched in https://github.com/envoyproxy/envoy/pull/33665 so @cainelli can you take a look?
https://github.com/envoyproxy/envoy/actions/runs/11193737190/job/31119411411
[ RUN      ] IpVersionsClientTypeDeferredProcessing/ExtProcIntegrationTest.GetAndCloseStreamWithTracing/IPv4_EnvoyGrpc_WithDeferredProcessing
test/extensions/filters/http/ext_proc/tracer_test_filter.cc:52: Failure
Expected equality of these values:
want
Which is: "0"
got
Which is: ""
grpc.status_code: 0 not found in tags:
component: proxy
status: canceled
upstream_address: ext_proc_server_0
upstream_cluster: ext_proc_server_0
Stack trace:
0x7f21ffe377ce: Envoy::Extensions::HttpFilters::ExternalProcessing::Span::~Span()
0x7f21ffe379a9: Envoy::Extensions::HttpFilters::ExternalProcessing::Span::~Span()
0x7f21954c5a7e: (unknown)
0x7f21954c59d0: (unknown)
0x7f219543405b: (unknown)
0x7f2195436300: Envoy::Grpc::AsyncStreamImpl::~AsyncStreamImpl()
0x7f2195436379: Envoy::Grpc::AsyncStreamImpl::~AsyncStreamImpl()
0x7f210c9e692e: (unknown)
0x7f210c969a60: (unknown)
0x7f210c957dfb: Envoy::Event::DispatcherImpl::clearDeferredDeleteList()
0x7f210c95f5fb: (unknown)
0x7f210c95f5a7: (unknown)
0x7f210c95f53f: (unknown)
0x7f210c95f4ff: (unknown)
0x7f210c95e12b: (unknown)
0x16b7ef6: (unknown)
0x16b7407: std::__1::function<>::operator()()
0x7f20ed6c5bd4: (unknown)
0x7f20ed6c5b89: (unknown)
0x1a0390c: (unknown)
0x19ff468: (unknown)
0x19fe56c: event_base_loop
0x7f20eda94b7e: Envoy::Event::LibeventScheduler::run()
0x7f210c9592fb: Envoy::Event::DispatcherImpl::run()
0x7f21bf8006f7: Envoy::Server::WorkerImpl::threadRoutine()
0x7f21bf808f2a: (unknown)
0x7f21bf808ec7: (unknown)
0x7f21bf808e5f: (unknown)
0x7f21bf808e1f: (unknown)
0x7f21bf807b7b: (unknown)
0x16b7ef6: (unknown)
0x16b7407: std::__1::function<>::operator()()
0x7f20d147df8a: (unknown)
0x7f20d147dee5: (unknown)
0x7f20c2988609: start_thread
test/integration/fake_upstream.cc:795: Failure
Failed
Please don't waitForHttpConnection with a 5s timeout if failure is expected
Stack trace:
0x7f21deab35c2: Envoy::FakeUpstream::waitForHttpConnection()
0x15cfc1b: (unknown)
0x140f139: (unknown)
0x7f20c338ea4d: testing::internal::HandleSehExceptionsInMethodIfSupported<>()
0x7f20c337522e: testing::internal::HandleExceptionsInMethodIfSupported<>()
0x7f20c335cb1d: testing::Test::Run()
0x7f20c335d60e: testing::TestInfo::Run()
... Google Test internal frames ...
test/extensions/filters/http/ext_proc/ext_proc_integration_test.cc:281: Failure
Value of: fake_upstreams_[0]->waitForHttpConnection(*dispatcher_, fake_upstream_connection_)
Actual: false (Timed out waiting for new connection.)
Expected: true
Stack trace:
0x15cfcb5: (unknown)
0x140f139: (unknown)
0x7f20c338ea4d: testing::internal::HandleSehExceptionsInMethodIfSupported<>()
0x7f20c337522e: testing::internal::HandleExceptionsInMethodIfSupported<>()
0x7f20c335cb1d: testing::Test::Run()
0x7f20c335d60e: testing::TestInfo::Run()
... Google Test internal frames ...
test/extensions/filters/http/ext_proc/ext_proc_integration_test.cc:277: Failure
Expected equality of these values:
std::to_string(status_code)
Which is: "200"
response.headers().getStatusValue()
Which is: "504"
Stack trace:
0x15d08a8: (unknown)
0x140f17c: (unknown)
0x7f20c338ea4d: testing::internal::HandleSehExceptionsInMethodIfSupported<>()
0x7f20c337522e: testing::internal::HandleExceptionsInMethodIfSupported<>()
0x7f20c335cb1d: testing::Test::Run()
0x7f20c335d60e: testing::TestInfo::Run()
... Google Test internal frames ...
[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_reloadable_features_no_extension_lookup_by_name to: true
[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_reloadable_features_defer_processing_backedup_streams to: true
[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_quic_always_support_server_preferred_address to: true
[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_reloadable_features_runtime_initialized to: false
[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_reloadable_features_no_extension_lookup_by_name to: true
[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_reloadable_features_defer_processing_backedup_streams to: true
[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_quic_always_support_server_preferred_address to: true
[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_reloadable_features_runtime_initialized to: false
[  FAILED  ] IpVersionsClientTypeDeferredProcessing/ExtProcIntegrationTest.GetAndCloseStreamWithTracing/IPv4_EnvoyGrpc_WithDeferredProcessing, where GetParam() = ((4-byte object <00-00 00-00>, 4-byte object <00-00 00-00>), true) (31694 ms)

@alyssawilk this is a duplicate of https://github.com/envoyproxy/envoy/issues/36041 which I need to dig into.


fix: import status window UX

Closes https://github.com/ipfs/ipfs-webui/issues/2030
Demo
(1st part of the demo is with mouse clicks, 2nd is with the keyboard navigation for accessibility)
https://user-images.githubusercontent.com/24696635/207647031-0ff21567-2c7e-4bfe-9cbd-5b6e7c90fe01.mov
Potential side-effects

I changed src/reset.css to use :focus-visible instead of :focus, which makes the outline box around "tabbable" elements only appear with KEYBOARD and not MOUSE interactions.

The demo looks great! Thanks so much for throwing this together. I'm going to check out the code now.


DRY up dependencies

Something like would DRY up the dependencies for a function-based skid: https://github.com/agrc/deq-eid-skid/commit/b3fb29b6e93869b4f8057e42e8de2c0d6bc1cc4f

I was wondering if you could read the requirements.txt file into the setup.py

That commit worked. I moved away from it when I switched to a dockerfile for cloudrun. No more need for the requirements file after that.

It's a nice improvement for function based skids still.


TestCafe Salesforce Automation - hammerhead Error "This page has an error. You might need to refresh it"

What is your Test Scenario?
Via automation tried to navigate to create firm/ create contact layout
What is the Current behavior?
Error on opening the page via test cafe
[Cannot read property 'hammerhead|event|internal-blur' of null]
_	cancelInternalEvents()@http://<IP_ADDRESS>:52557/hammerhead.js:13:27910
{anonymous}()@http://<IP_ADDRESS>:52557/hammerhead.js:12:6
HTMLElement.dispatchEvent()@http://<IP_ADDRESS>:52557/hammerhead.js:13:27061
me.handleBlur()@https://preqin--uat.lightning.force.com/components/lightning/input.js:1:35128
What is the Expected behavior?
The page should load successfully and the user able to add the details
What is your web application and your TestCafe test code?

https://preqin--uat.lightning.force.com/
Debug Logs:
[Cannot read property 'hammerhead|event|internal-blur' of null]
_	cancelInternalEvents()@http://<IP_ADDRESS>:52557/hammerhead.js:13:27910
{anonymous}()@http://<IP_ADDRESS>:52557/hammerhead.js:12:6
HTMLElement.dispatchEvent()@http://<IP_ADDRESS>:52557/hammerhead.js:13:27061
me.handleBlur()@https://preqin--uat.lightning.force.com/components/lightning/input.js:1:35128

import { Selector } from 'testcafe';
fixture New Fixture
.page Env Url;
test('New Test', async (t: TestController) => {
await t
.click('#username')
.typeText('#username', '*')
.click('#password')
.typeText('#password', '')
.click('#Login')
.click(Selector('button').withText('Show Navigation Menu'))
.click(Selector('#navMenuList span').withText('Firms').nth(1))
.click(Selector('#brandBand_1 div').withText('New').nth(7))
.click(Selector('#brandBand_2 span').withText('Next'))
.wait(8000);
});

Screenshots:

Steps to Reproduce:

Go to my website ...
Navigate to FIrm/Contact page
Click on new
See the error (Viewed only via automation)

Your Environment details:

testcafe version:                   <1.15.2>
node.js version:                    <v12.18.2>
command-line arguments:
browser name and version: <Version 92.0.4515.131>
platform and version:          < Windows>
other:

@ArunjaiAcc
Hello,
We would love to assist you and diagnose this issue. However, since your website requires authentication, I will not be able to access it. Our policy prevents us from accessing a customer’s internal resources without prior written approval from the entity that owns the server/web resource. If you want us to research the problem further, we’ll need access to the server/web resource and your account using your credentials. Please send us<EMAIL_ADDRESS>written confirmation. It must permit DevExpress personnel to remotely access your website account and its internal resources for research/testing/and debugging purposes.
Could you please send the complete test code with credentials and the written confirmation to our mailbox<EMAIL_ADDRESS>

Sure, I will share the same @Farfurix

@ArunjaiAcc , please leave a comment here after you send the confirmation to<EMAIL_ADDRESS>

I have shared the detail with<EMAIL_ADDRESS>Please check into this @aleks-pro

@ArunjaiAcc
Hello,
We couldn't find your Email. Would you please re-send it? How do we identify it?

Hi Team,
Please find the attached source code for automation.
Regards,
Arunkumar
On Wed, Aug 25, 2021 at 1:03 PM Vladimir Airikh @.***>
wrote:

@ArunjaiAcc https://github.com/ArunjaiAcc
Hello,
We couldn't find your Email. Would you please re-send it? How do we
identify it?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/DevExpress/testcafe/issues/6459#issuecomment-905256414,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AKXAVPHSGRPXYNSCNOEA5Q3T6SMD5ANCNFSM5CAUZ6FQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email
.

Hello @Farfurix  I re-sended it

import { Selector } from 'testcafe';
fixture New Fixture
.page Env Url;
test('New Test', async (t: TestController) => {
await t
.click('#username')
.typeText('#username', '*')
.click('#password')
.typeText('#password', '')
.click('#Login')
.click(Selector('button').withText('Show Navigation Menu'))
.click(Selector('#navMenuList span').withText('Firms').nth(1))
.click(Selector('#brandBand_1 div').withText('New').nth(7))
.click(Selector('#brandBand_2 span').withText('Next'))
.wait(8000);
});

@ArunjaiAcc
Hello,
I reproduced this issue by clicking the following drop-down menu:

We will update this thread once we have any news.
As a workaround, you can use another way to reach the "New Firms" page, without clicking a drop-down menu:

Click the "Firms" tab
Click the "New" button

Thanks, I'll check it out.
On Wed, Sep 1, 2021 at 2:02 PM Vladimir Airikh @.***>
wrote:

@ArunjaiAcc https://github.com/ArunjaiAcc
Hello,
I reproduced this issue by clicking the following drop-down menu:
[image: Capture]
https://user-images.githubusercontent.com/30019338/131634838-92009ef6-5727-410f-ad2b-7b48985de82e.PNG
We will update this thread once we have any news.
As a workaround, you can use another way to reach the "New Firms" page,
without clicking a drop-down menu:

Click the "Firms" tab
Click the "New" button

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/DevExpress/testcafe/issues/6459#issuecomment-910063736,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AKXAVPF7MVOUXQL5QOX5HELT7XQJFANCNFSM5CAUZ6FQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Hi that work around doesn’t work. I have the same issue only I browse directly to the page and it still fails. This has been broken since June, is there a plan to resolve this issue?

Hi @aozomaro,
Please share your example for further research. There must be another issue.

Hello @ArunjaiAcc,
I've just tried to reproduce this issue with<EMAIL_ADDRESS>However, the testing URL you shared is no longer available.
Could you try running your test with<EMAIL_ADDRESS>with the experimental-proxyless option enabled? This option enables native browser automation. In Proxyless mode, a few issues are already fixed. This issue might also be fixed in Proxyless mode.
This option is available in all interfaces:
// Command-line
testcafe chrome tests --experimental-proxyless

// Programmatic
const testcafe = await createTestCafe({ experimentalProxyless: true });

// Configuration file
{
"experimentalProxyless": "true"
}

Please keep in mind that this mode is experimental. Also, the Proxyless mode is implemented only in Google Chrome. It will not work correctly if you run tests in a non-Chrome browser or in a combination of other browsers.

Hi,
Sure, I will check and let you know.
Regards,
Arunkumar
On Mon, Feb 6, 2023 at 7:36 PM Artem-Babich @.***>
wrote:

Hello @ArunjaiAcc https://github.com/ArunjaiAcc,
I've just tried to reproduce this issue with @.***
However, the testing URL you shared is no longer available.
Could you try running your test with @.*** with the experimental-proxyless
option
https://testcafe.io/documentation/404237/guides/experimental-capabilities/proxyless-mode
enabled? This option enables native browser automation. In Proxyless mode,
a few issues are already fixed. This issue might also be fixed in Proxyless
mode.
This option is available in all interfaces:
// Command-line
testcafe chrome tests --experimental-proxyless
// Programmatic
const testcafe = await createTestCafe({ experimentalProxyless: true });
// Configuration file
{
"experimentalProxyless": "true"
}
Please keep in mind that this mode is experimental. Also, the Proxyless
mode is implemented only in Google Chrome. It will not work correctly if
you run tests in a non-Chrome browser or in a combination of other browsers.
—
Reply to this email directly, view it on GitHub
https://github.com/DevExpress/testcafe/issues/6459#issuecomment-1419135816,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AKXAVPA4MZPPV3BWHAYSSFDWWEAOXANCNFSM5CAUZ6FQ
.
You are receiving this because you were mentioned.Message ID:
@.***>


Update/readme

Clarify which yarn to use and the usage of command.

I actually don't understand what the command $ yarn (without the flag "start <data-file-path") is doing. It it does not generate any real data, then we should remove it.

@nanjiangshu Thanks for the PR. The yarn command is short for yarn install. It install the NPM packages under node_modules.

Thanks @e0 for clarification! So there are two steps,
The command $ yarn installs NPM packages and then
 $ yarn start <path-of-data-file> generates data.

So there are two steps

That's exactly right, we were way too sloppy on the documentation before.

As suggested by @e0 it would be good to add the minimum versions required too.

This Readme assumed a lot of prior knowledge, so it's really good to address this. Let's also clarify that while these commands can be run on their own, the output is meaningful only for the deployment pipeline.
Here is the documentation for the yarn command. It would be good to have this in the Readme too.

This comment is addressed.


ARROW-1619: [Java] Set lastSet in JsonFileReader

When reading a vector in JsonFileReader, lastSet should be set in VariableWidthVectors after reading inner vectors or else subsequent operations could corrupt the offsets.  This also allows to simplify some of the related code.  Additionally, ListVector.lastSet should be explicitly initialized to 0, which is it's starting offset.

cc @icexelloss @siddharthteotia

@icexelloss would you be ok with this for now and I'll put the todo comment back in to look into using loadFieldBuffers for future reference?

Yes I think that's fair.

@BryanCutler can you add a test for this issue?

can you add a test for this issue?

I think the real test for this is that you can call setValueCount on the vector at the end and it won't fill it with empty values for the VarChar/Binary cases or corrupt the offset vector for ListVector.  The existing JSON tests cover this pretty well, but looks like there's nothing for VarBinaryVector so I can add that.

I think the real test for this is that you can call setValueCount on the vector at the end and it won't fill it
with empty values for the VarChar/Binary cases or corrupt the offset vector for ListVector. The existing JSON tests cover this pretty well, but looks like there's nothing for VarBinaryVector so I can add that.

Sorry I am a bit confused. The current TestJSONFile doesn't cover this bug, right?

If you remove the calls to setLastSet here, a number of the tests fail that use ListVector and VarCharVector.  It's because setValueCount for these will modify the values/offsets according to the what lastSet is.

Thanks @BryanCutler !

Thanks @BryanCutler and @icexelloss! merging


Can't set properties in 'Column' element

The component doesn't follow the the props.

found the issue, i was passing in a string instead on an integer


Update package.json to allow for local Ceramic daemon config

Current use-case allows for the shipped composedb.config.json file to be used.  This is great if I'm starting with this tool, but if I have done some data modeling outside of this then I probably already have a Ceramic node instance I would like to use.
This PR adds a script that can be run to specify this behavior.

I understand the desire for this to be implemented, but I think this implementation is a little too isolated. My concern might not be one that's shared however this is intended to require minimal tinkering with the configuration files by the developer, adding a script that does not pass a configuration file in won't fully solve the ability to run the node locally without changing the run script as well.
I think I would rather see the change made to the run script to parse a flag like --local or something similar using process.argv and removing the config flag from the package file and adding it to the run script. I hope that this makes sense.


branch-2.0 fix data lost when FE leader change

Proposed changes
#33678

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


Webservice, send seed in header

Although the webservice will be removed from the projects (see #2), currently I want a way to get the seed which was used to generate the image.
I want a header in the style of:
X-Seed: 'foobar'

Implemented in 53215bd8ddabcebee34df5f2fed5e02b761b05cc.


chore(deps): upgrade project dependencies

Upgrade all dependencies except coverage. At the moment, coverage can't be upgraded to a version higher than 6.x as coveralls-python - a dependency of this project - has an upper bound on the supported coverage version. See this issue for details. Moving forward, we should consider dropping coveralls-python as a dependency since it no longer appears to be maintained.

:tada: This PR is included in version 1.0.0-rc.1 :tada:
The release is available on:

v1.0.0-rc.1
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 1.0.0 :tada:
The release is available on:

v1.0.0
GitHub release

Your semantic-release bot :package::rocket:


allow defining a vault password file

Issue Type

Feature request

Molecule and Ansible details
ansible --version

2.5.2

molecule --version

2.13.1

Molecule installation method (one of):

pip

Ansible installation method (one of):

pip

Desired Behavior
I want to be able to define a file containing a vault password in the provisioner section.
This should also set the necessary entry in the ansible.cfg file that molecule generates, so it is possible to use ansible-vault inside plays.
Actual Behaviour (Bug report only)
#1061 states that it would be possible to pass this as runtime parameter (it is also possible to pass an absolute path via config_options). This works a bit, but won't work when calling ansible-vault from within a play.
An alternative would be to copy the password file to the folder in /tmp too, so it is possible to refer to it by a relative path. This could be extended to allow putting any random file(s) there that should be referenced in plays.

This unfortunately doesn't work (the variable isn't used/expanded by molecule):
provisioner:
  name: ansible
  config_options:
    defaults:
      vault_password_file: ${MOLECULE_SCENARIO_DIRECTORY}/vault.pw
  lint:
    name: ansible-lint

It would be great if the vault.pw file (or any other file I define) could end up in MOLECULE_EPHEMERAL_DIRECTORY

This should be configured via molecule.yml as:
provisioner:
  name: ansible
  options:
    'vault-id'<EMAIL_ADDRESS>
The command will then become:
/Users/jodewey/.pyenv/versions/molecule2-env-2.7.13/bin/ansible-playbook<EMAIL_ADDRESS>--skip-tags=molecule-notest,notest --inventory=/var/folders/_j/ncm1yw111bb771f6zsw1d3lc0000gn/T/molecule/docker/default/ansible_inventory.yml --diff /Users/jodewey/git/molecule_2/test/scenarios/driver/docker/molecule/default/playbook.yml -vvv```

It would be great if the vault.pw file (or any other file I define) could end up in MOLECULE_EPHEMERAL_DIRECTORY or the above would work.

I agree this would be a nice to have, but isn't trivial.  It's a bit of a chicken and egg problem.  Molecule needs to parse the config to then construct all the pathing for most of the env vars exported by Molecule.  I'd like to do it at some point, but it's not something we will get to now.

Closing as can be fixed by #1287.

I agree this would be a nice to have, but isn't trivial. It's a bit of a chicken and egg problem. Molecule needs to parse the config to then construct all the pathing for most of the env vars exported by Molecule. I'd like to do it at some point, but it's not something we will get to now.

FYI - #1296

FYI for those who are bumping against this, I'm using molecule 2.22, there it's possible to specify it as a config option:
provisioner:
  name: ansible
  config_options:
    defaults:
      vault_password_file: ~/.ansible-vault-password-file.txt

I'm looking for the same for vault_identity_list.  Which works properly inside the local ansible.cfg
In, molecule.yml, I have
  name: ansible
  options:
    vvv: true
  config_options:
    default:
      vault_identity_list<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
This yields:
$ cat ~/.cache/molecule/workstation/default/ansible.cfg
# Molecule managed

[defaults]
ansible_managed = Ansible managed: Do NOT edit this file manually!
display_failed_stderr = True
forks = 50
retry_files_enabled = False
host_key_checking = False
nocows = 1
interpreter_python = auto
[ssh_connection]
scp_if_ssh = True
control_path = %(directory)s/%%h-%%p-%%r
[default]
vault_identity_list =<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
But I still get:
ERROR! Attempting to decrypt but no vault secrets found
ERROR:

Version info:
$ molecule --version
molecule 3.0.6
   ansible==2.9.10 python==3.8

You just misspelled defaults in your molecule.yml.


Input reading isn't constrained by max open file count

If you have something like 512 CPUs but can only have 1024 open files, you immediately get
/tmp/geom.XXCqXw0m: Too many open files

I think this is because read_input opens a set of temporary files for each CPU without checking whether that will run the process out of files.

https://github.com/mapbox/tippecanoe/pull/396 should fix this.

I use tippecanoe to process 771 geojson files (104G) on a 16-cores machine, it complains "/tmp/geom.XXs1HFBW: Too many open files".

Thanks for the report. I am surprised that the number of input files could make a difference, because they should be read one at a time. What options are you using? Does it work with smaller numbers of input files? What limits does ulimit -a show?

Thanks. It looks like Tippecanoe is leaving input files open at the end of reading when using --read-parallel. I'll figure out what's going wrong and fix it.

@ericfisher Thanks for help. Is it normal or expected that reading a bunch of geojson files (104 GB, tens of millions small polygons) would spend a whole day?

That doesn't seem completely unreasonable to me. I don't have a lot of benchmarks for data of that size, but complete census blocks for the United States (about 11 million polygons) take about 13 hours of CPU time to read and tile.
As far as I can tell, the time it takes Tippecanoe to parse input is roughly comparable to how long it takes other tools like ogr2ogr to read the same files. If you see any big discrepancies, it would be useful to have examples of files that are especially slow so I can address those cases.
Reading Geobuf is faster than reading GeoJSON, so it might help to convert your files to Geobuf, but ultimately I think you would spend as much time on the conversion as you would save on reading the result into Tippecanoe.

The files being left open will be fixed in https://github.com/mapbox/tippecanoe/pull/529

Merged


when update a hashs by redis.hset("rrr", "item5", "value3") will return false

redis.hset("rrr", "item5", "value3")  when first set it, return true
but redis.hset("rrr", "item5", "value4") return false and the redis server was update the value to "value4"

According to official redis-docs it returns 0 if field already exists and value was updated

@rtenagom  so, why not return ture? return 0 is successful in c and redis


pass fd to PackageDescription while parsing manifests instead of using SWIFT_DUMP_PACKAGE

Looks good. I am currently waiting on the swift-3-api-guidelines merge before merging anything new.

@swift-ci please test

I'm going to start merging everything, as I hoped the swift 3 apis would be merged this week, but it looks like it will be longer.

@swift-ci Please test

@swift-ci Please test

/home/buildnode/jenkins/workspace/swift-package-manager-PR-Linux/swiftpm/Sources/ManifestParser/Manifest+parse.swift:74:13: error: use of unresolved identifier 'fclose'
    defer { fclose(fp) }
            ^~~~~~
<unknown>:0: error: build had 1 command failures

This did not error for me on OSX and Ubuntu 15.10
CI looks like to be on 14.04, will check.

This is weird, It works for me on Ubuntu 14.04 too.
Latest toolchain
Swift version 3.0-dev (LLVM b361b0fc05, Clang 11493b0f62, Swift 24a0c3de75)
Target: x86_64-unknown-linux-gnu

Probably would be fixed if you import func libc.fclose

I figured but weird that CI errors out where as nothing on local

@mxcl trigger CI please

@swift-ci Please test

CI succeeded merging


Epic: Text Analytics 1.0.0-preview.2

Tracking epic for preview2 items:

[x] #6932 Clean up samples
[x] #6933 Add recorded tests
[x] #6934 Add live tests
[x] #6970 Allow credential rolling
[x] #6971 Improve error model
[x] #7055 Respond to feedback

Preview 2 released!


Workaround for Jaeger 1468 create traces so index cleaning doesn't fail

Signed-off-by: Kevin Earls<EMAIL_ADDRESS>This is a workaround for https://github.com/jaegertracing/jaeger/issues/1468  It creates at least one trace so that the es-index-cleaner jobs will not fail on account of there not being any indices.

@objectiser  @pavolloffay  Please review

Good point.  I guess there is a race condition here, and this way the first trace gets created faster, so it's there the first time the es-index-cleaner runs?  Let me see if I can find another solution.


Missing demo resources

Hi Peter
Many of the demos result in black screen. This is from developer console (Chrome):
simple.html:25 Uncaught ReferenceError: Subscene3D is not defined
Some end up with a different problem, pathtube for example.
glutil_min.js:92 Uncaught TypeError: this.bufferedMesh._getBounds is not a function
Thank you for the great work BTW.
Srdjan

I’m in the process of decoupling textures, meshes, shapes, etc. from WebGL resources.  Subscene3D is among the new classes created for this purpose.  Not all the demos have been modified to match yet. If you want to try this library, see the Releases page (however, the classes and functionality they use are likely to be deprecated in the next version.)
From: Srdjan Dakic
Sent: Wednesday, May 11, 2016 2:13 AM
To: peteroupc/html3dutil
Subject: [peteroupc/html3dutil] Missing demo resources (#6)
Hi Peter
Many of the demos result in black screen. This is from developer console (Chrome):
simple.html:25 Uncaught ReferenceError: Subscene3D is not defined
Some end up with a different problem, pathtube for example.
glutil_min.js:92 Uncaught TypeError: this.bufferedMesh._getBounds is not a function
Thank you for the great work BTW.
Srdjan
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub


Use Powermock annotation to support javax.crypto

This change fixes the failing tests in AlluxioMasterProcessTest. See this SO post for details about the solution.

Codecov Report

Merging #14538 (de0bdc8) into master (c642abc) will decrease coverage by 18.66%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##             master   #14538       +/-   ##
=============================================
- Coverage     41.84%   23.17%   -18.67%
+ Complexity     9331     4294     -5037
=============================================
  Files          1494     1494
  Lines         87668    87681       +13
  Branches      10449    10453        +4
=============================================
- Hits          36682    20320    -16362
- Misses        48004    65646    +17642
+ Partials       2982     1715     -1267

Impacted Files
Coverage Δ

...mon/src/main/java/alluxio/shell/CommandReturn.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...mon/src/main/java/alluxio/util/ExceptionUtils.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...n/src/main/java/alluxio/wire/AlluxioProxyInfo.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../src/main/java/alluxio/wire/AlluxioMasterInfo.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...src/main/java/alluxio/job/meta/JobIdGenerator.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...n/src/main/java/alluxio/stress/BaseParameters.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...main/java/alluxio/underfs/options/ListOptions.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...main/java/alluxio/worker/block/io/BlockReader.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...main/java/alluxio/worker/block/io/BlockWriter.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...mon/src/main/java/alluxio/table/common/Layout.java
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 845 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c642abc...de0bdc8. Read the comment docs.

alluxio-bot, merge this please

alluxio-bot, cherry-pick this to branch-2.7 please


AssertionError install tests for 0.2.9 build on NixOS

Hi Team,
I was trying to enable the test cases for pythonPackages.imgaug https://github.com/NixOS/nixpkgs/pull/67494
During this process i am able to execute the test cases but facing AssertionError and this is causing 5 failures.
Summary of test run:
============ **5 failed, 383 passed, 3 warnings in 199.71s (0:03:19)** =============
detailed log :
imgaug_test_failures.txt
Please suggest. Thanks.

The first and second test are now a little bit looser in master and should run through. The float128(value) was already in master. So, in theory all tests should now succeed.

The first and second test are now a little bit looser in master and should run through. The float128(value) was already in master. So, in theory all tests should now succeed.

Thanks. I will align my code to these changes and test.

The first and second test are now a little bit looser in master and should run through. The float128(value) was already in master. So, in theory all tests should now succeed.

Hi @aleju ,
After moving to latest commit from 0.2.9, I am getting new error message while running install tests
pytest: error: unrecognized arguments: --xdoctest
running install tests ERROR: usage: pytest [options] [file_or_dir] [file_or_dir] [...] pytest: error: unrecognized arguments: --xdoctest    inifile: /build/source/pytest.ini    rootdir: /build/source
Please suggest. Thanks.

What command are you executing? I assume it contains --xdoctest-modules.
You need the xdoctest package to run the tests with --xdoctest-modules, which verifies that all examples in the library's docstrings run without errors and produce the expected outputs. You can however remove the --xdoctest-modules argument, then only the unittests are run, which should be enough to verify that the library works correctly.

Hi @aleju ,
initially i was just using pytest command.
now I have modified the pytest.ini
from
addopts = -p pytester -p no:doctest --xdoctest --xdoctest-global-exec="import imgaug as ia\nfrom imgaug import augmenters as iaa"
to
addopts = -p pytester -p no:doctest"
### But now it is stuck after 79%
`running install tests
============================= test session starts ==============================
platform linux -- Python 3.7.4, pytest-5.1.0, py-1.8.0, pluggy-0.12.0
rootdir: /build/source, inifile: pytest.ini
collected 3097 items
test/test_dtypes.py .................................................... [  1%]
...................................................................      [  3%]
test/test_imgaug.py .................................................... [  5%]
....                                                                     [  5%]
test/test_multicore.py ............................                      [  6%]
test/test_parameters.py ................................................ [  8%]
........................................................................ [ 10%]
........................................................................ [ 12%]
........................................................................ [ 15%]
.......................................                                  [ 16%]
test/test_random.py .................................................... [ 18%]
........................................................................ [ 20%]
...........................                                              [ 21%]
test/augmentables/test_batches.py ......                                 [ 21%]
test/augmentables/test_bbs.py .......................................... [ 22%]
..............................................................           [ 24%]
test/augmentables/test_heatmaps.py .........................             [ 25%]
test/augmentables/test_kps.py .......................................... [ 26%]
...............................                                          [ 27%]
test/augmentables/test_lines.py ........................................ [ 29%]
........................................................................ [ 31%]
........................................................................ [ 33%]
..........................                                               [ 34%]
test/augmentables/test_normalization.py ......................           [ 35%]
test/augmentables/test_polys.py ........................................ [ 36%]
........................................................................ [ 39%]
........................................................................ [ 41%]
...................................................................      [ 43%]
test/augmentables/test_segmaps.py ...................................... [ 44%]
...                                                                      [ 44%]
test/augmentables/test_utils.py ...................                      [ 45%]
test/augmenters/test_arithmetic.py ..................................... [ 46%]
........................................................................ [ 48%]
......................................                                   [ 50%]
test/augmenters/test_blend.py .......................................... [ 51%]
....................................                                     [ 52%]
test/augmenters/test_blur.py ........................................... [ 54%]
.........                                                                [ 54%]
test/augmenters/test_color.py .......................................... [ 55%]
........................................................                 [ 57%]
test/augmenters/test_contrast.py ....................................... [ 58%]
...........................                                              [ 59%]
test/augmenters/test_convolutional.py .................................. [ 60%]
...........                                                              [ 61%]
test/augmenters/test_edges.py ......................                     [ 61%]
test/augmenters/test_flip.py ........................................... [ 63%]
...............................................                          [ 64%]
test/augmenters/test_geometric.py ...................................... [ 65%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 75%]
.............                                                            [ 75%]
test/augmenters/test_meta.py ........................................... [ 77%]
........................................................................ [ 79%]
`
Please suggest.
Thanks.

I usually use the command python -m pytest test/ --verbose --xdoctest-modules -s --durations=20 which shows exactly which tests were executed, not just the files.
I can't tell exactly where it hangs as there is no corresponding output. Even if there was, it would probably be hard to debug it from the distance.
My guess would be that it is somehow connected to multiprocessing (though it is weird that the tests of multicore.py all passed). I would try commenting out the tests test_augment_batches_background and test_augment_batches_with_many_different_augmenters in test_meta.py and see if it then works. But that would just show in which test it fails, not in which line and especially what the root cause of the issue is.

Hi @aleju , Please find last output of imgaug unit test.
imgaug_nix_hang_state.txt
test/augmenters/test_meta.py::TestAugmenter::test_localize_random_state PASSED test/augmenters/test_meta.py::TestAugmenter::test_reseed PASSED test/augmenters/test_meta.py::TestAugmenter::test_reseed_deterministic_too PASSED test/augmenters/test_meta.py::TestAugmenter::test_reseed_with_rng PASSED test/augmenters/test_meta.py::TestAugmenter::test_reseed_with_seed PASSED test/augmenters/test_meta.py::TestAugmenter_augment_batches::test_augment_batches_background PASSED test/augmenters/test_meta.py::TestAugmenter_augment_batches::test_augment_batches_batch_with_array_of_images PASSED test/augmenters/test_meta.py::TestAugmenter_augment_batches::test_augment_batches_batch_with_list_of_images PASSED test/augmenters/test_meta.py::TestAugmenter_augment_batches::test_augment_batches_invalid_datatype PASSED test/augmenters/test_meta.py::TestAugmenter_augment_batches::test_augment_batches_list_of_arrays_deprecated PASSED test/augmenters/test_meta.py::TestAugmenter_augment_batches::test_augment_batches_list_of_empty_list_deprecated PASSED test/augmenters/test_meta.py::TestAugmenter_augment_batches::test_augment_batches_list_of_invalid_datatype PASSED test/augmenters/test_meta.py::TestAugmenter_augment_batches::test_augment_batches_list_of_list_of_arrays_deprecated PASSED test/augmenters/test_meta.py::TestAugmenter_augment_batches::test_augment_batches_list_of_list_of_invalid_datatype PASSED
Please note that we donot have support for xdoctest on NixOS at present so I am removing     --xdoctest-modules from above command.
Thanks.

Hm, I thought that command showed the name of test currently being executed, but apparently it does so only after the test has fully run through. It also doesn't follow the order in which the tests were defined in the file. That makes things a bit harder. At least I can see that it fails at TestAugmenter_augment_batches, which is what I suspected initially. It succeeds with test_augment_batches_background, but test_augment_batches_with_many_different_augmenters is not shown as passed, which makes that test the most likely cause of the problem.
I would try now python -m pytest ./test/augmenters/test_meta.py::TestAugmenter_augment_batches::test_augment_batches_with_many_different_augmenters --verbose --xdoctest-modules -s --durations=20 to only exectute the mentioned test and see if that command hangs. If that is the case, I would change within the test test_augment_batches_with_many_different_augmenters in test/augmenters/test_meta.py
        for aug in augs:
            nb_changed = 0

to
        for aug in augs:
            print(aug.name)
            nb_changed = 0

and then run the above pytest command again. It should usually print something. In case it doesn't, it would mean that it only prints once the test fully ran through, which it never does here. In that case I would start with the following steps below.
Step 1:
Remove the block
            iaa.Lambda(
                func_images=lambda images, random_state, parents, hooks: images,
                func_keypoints=lambda keypoints_on_images, random_state,
                parents, hooks: keypoints_on_images,
                name="Lambda-nochange"
            ),
            iaa.AssertLambda(
                func_images=lambda images, random_state, parents, hooks: True,
                func_keypoints=lambda keypoints_on_images, random_state,
                parents, hooks: True,
                name="AssertLambda-nochange"
            ),

from the test. Lambda functions can sometimes cause problems in connections with multiprocessing. Try the pytest command again.
Step 2:
If the test still hangs, I would then also remove iaa.Convolve(...), iaa.Sharpen(...), iaa.Emboss(...), iaa.EdgeDetect(...) and iaa.DirectedEdgeDetect(...). In older commits of master these still use lambda functions in the background and could also cause issues. Try pytest again.
Step 3:
Try to remove the remaining augmenters one by one, running pytest again each time, until the culprit is found.
Btw, how long do you wait for the test to finish? It does take a few seconds on my machine.

Hm, I thought that command showed the name of test currently being executed, but apparently it does so only after the test has fully run through. It also doesn't follow the order in which the tests were defined in the file. That makes things a bit harder. At least I can see that it fails at TestAugmenter_augment_batches, which is what I suspected initially. It succeeds with test_augment_batches_background, but test_augment_batches_with_many_different_augmenters is not shown as passed, which makes that test the most likely cause of the problem.

Hi @aleju ,
I have taken latest 0.3.0 changes.
As per my observation the execution order is alphabetic here.
Yes, it is  test_augment_batches_with_many_different_augmenters  is the one which is stuck.
But behaviour is different when executed differently.
I mean if try to run
test/augmenters/test_meta.py::TestAugmenter_augment_batches::test_augment_batches_with_many_different_augmenters - Pass
test/augmenters/test_meta.py::TestAugmenter_augment_batches - Pass
test/augmenters/test_meta.py -  Pass
Pass logs : imgaug_test_meta_py_pass.txt
test/augmenters -  Hangs on test_augment_batches_with_many_different_augmenters based on assumption that tests are run in alphabetic order.
I had waited for more than 1 hour and test is still stuck.
Please suggest.
Thanks.

Thanks.

I had waited for more than 1 hour and test is still stuck.

Yes, that is way beyond the expected timeframe. On my machine it takes a few seconds. On a slow laptop maybe 30s.
The observed hanging is very weird. Among all test files, I think only test_multicore.py and test_meta.py use the multiprocessing module and latter one is not run in your case. So there shouldn't be anything messing up the multiprocessing stuff. Most augmenters also don't cache any data. Only very few do and the cached data consists only of numpy arrays. No explicit cleanup should be necessary there either... That increases the probability that it is somehow caused by something that numpy or OpenCV do internally or by NixOS behaving slightly different compared to what python's multiprocessing expects.
Some things I would do here to pinpoint the problem:

Run each of the test commands that you mentioned a handful of times to ensure that the results are consistent.
In the command that you ran that hangs, move test_meta.py one step earlier in the list of executed files and see if the command then succeeds. Repeat until it succeeds. Run each command a handful of times to exclude that its up to random chance when it hangs.
In test_meta.py, function test_augment_batches_with_many_different_augmenters() in list augs = [(...)] comment out all augmenters and see if the test succeeds. Then reactivate each one of them one by one until it fails and see which augmenter that is. Also then check if that is the only augmenter that makes it fail.
In test_meta.py in test_augment_batches_with_many_different_augmenters() add import cv2; cv2.setNumThreads(1) at the very start of the function and see if the hanging stops.
Upgrade numpy to 1.17 if that hasn't been done yet. Otherwise downgrade from 1.17 to 1.16 and see if it works then. 1.17 introduced various changes to the random number sampling.

I have now seen similar behaviour. In my case it was caused by an exception in a subprocess that for some reason was not raised into the main process, even though multiprocessing should do that. ctrl+c showed the error however.
Can you copy-paste the following code block into a file, run it and post the output. If it hangs again, ctrl+c and post the full stacktrace -- it might contain two separate errors.
import six.moves as sm
import numpy as np
import imgaug as ia
import imgaug.augmenters as iaa

for background in [False, True]:
    image = np.array([[0, 0, 1, 1, 1],
                      [0, 0, 1, 1, 1],
                      [0, 1, 1, 1, 1]], dtype=np.uint8)
    keypoint = ia.Keypoint(x=2, y=1)
    keypoints = [ia.KeypointsOnImage([keypoint], shape=image.shape + (1,))]

    def _lambda_func_images(images, random_state, parents, hooks):
        return images

    def _lambda_func_keypoints(keypoints_on_images, random_state,
                               parents, hooks):
        return keypoints_on_images

    def _assertlambda_func_images(images, random_state, parents, hooks):
        return True

    def _assertlambda_func_keypoints(keypoints_on_images, random_state, parents, hooks):
        return True

    augs = [
        iaa.Sequential([iaa.Fliplr(1.0), iaa.Flipud(1.0)]),
        iaa.SomeOf(1, [iaa.Fliplr(1.0), iaa.Flipud(1.0)]),
        iaa.OneOf([iaa.Fliplr(1.0), iaa.Flipud(1.0)]),
        iaa.Sometimes(1.0, iaa.Fliplr(1)),
        iaa.WithColorspace("HSV", children=iaa.Add((-50, 50))),
        iaa.WithChannels([0], iaa.Add((-50, 50))),
        iaa.Noop(name="Noop-nochange"),
        iaa.Lambda(
            func_images=_lambda_func_images,
            func_keypoints=_lambda_func_keypoints,
            name="Lambda-nochange"
        ),
        iaa.AssertLambda(
            func_images=_assertlambda_func_images,
            func_keypoints=_assertlambda_func_keypoints,
            name="AssertLambda-nochange"
        ),
        iaa.AssertShape(
            (None, 64, 64, 3),
            check_keypoints=False,
            name="AssertShape-nochange"
        ),
        iaa.Resize((0.5, 0.9)),
        iaa.CropAndPad(px=(-50, 50)),
        iaa.Pad(px=(1, 50)),
        iaa.Crop(px=(1, 50)),
        iaa.Fliplr(1.0),
        iaa.Flipud(1.0),
        iaa.Superpixels(p_replace=(0.25, 1.0), n_segments=(16, 128)),
        iaa.ChangeColorspace(to_colorspace="GRAY"),
        iaa.Grayscale(alpha=(0.1, 1.0)),
        iaa.GaussianBlur(1.0),
        iaa.AverageBlur(5),
        iaa.MedianBlur(5),
        iaa.Convolve(np.array([[0, 1, 0],
                               [1, -4, 1],
                               [0, 1, 0]])),
        iaa.Sharpen(alpha=(0.1, 1.0), lightness=(0.8, 1.2)),
        iaa.Emboss(alpha=(0.1, 1.0), strength=(0.8, 1.2)),
        iaa.EdgeDetect(alpha=(0.1, 1.0)),
        iaa.DirectedEdgeDetect(alpha=(0.1, 1.0), direction=(0.0, 1.0)),
        iaa.Add((-50, 50)),
        iaa.AddElementwise((-50, 50)),
        iaa.AdditiveGaussianNoise(scale=(0.1, 1.0)),
        iaa.Multiply((0.6, 1.4)),
        iaa.MultiplyElementwise((0.6, 1.4)),
        iaa.Dropout((0.3, 0.5)),
        iaa.CoarseDropout((0.3, 0.5), size_percent=(0.05, 0.2)),
        iaa.Invert(0.5),
        iaa.Affine(
            scale=(0.7, 1.3),
            translate_percent=(-0.1, 0.1),
            rotate=(-20, 20),
            shear=(-20, 20),
            order=ia.ALL,
            mode=ia.ALL,
            cval=(0, 255)),
        iaa.PiecewiseAffine(scale=(0.1, 0.3)),
        iaa.ElasticTransformation(alpha=0.5)
    ]

    nb_iterations = 100
    image = ia.quokka(size=(64, 64))
    batches = [ia.Batch(images=[np.copy(image)],
                        keypoints=[keypoints[0].deepcopy()])
               for _ in sm.xrange(nb_iterations)]
    for aug in augs:
        print("[bg=%s] Running %s..." % (background, aug.name))
        nb_changed = 0
        batches_aug = list(aug.augment_batches(batches, background=background))
        for batch_aug in batches_aug:
            image_aug = batch_aug.images_aug[0]
            if (image.shape != image_aug.shape
                    or not np.array_equal(image, image_aug)):
                nb_changed += 1
                if nb_changed > 10:
                    break
        if "-nochange" not in aug.name:
            assert nb_changed > 0
        else:
            assert nb_changed == 0

Hi @aleju ,
For me it is stuck at :
---- [bg=True] Running UnnamedChangeColorspace... [bg=True] Running UnnamedGrayscale... [bg=True] Running UnnamedGaussianBlur...
img_aug_debug.txt
Thanks.

@aleju Any news on this?

Sorry, didn't have much time the last couple of days, so this issue was delayed a bit.
Based on the outputs my best guess is that OpenCV causes issues when being executed in a subprocess, as the method that hangs uses OpenCV. (Which would also mean that any user trying to call OpenCV in a subprocess might encounter this issue, even without using imgaug.)
I have seen something like that before and it was fixed using cv2.setNumThreads(0). Can you try the two code blocks below and see if any of them works? Also, did ctrl+c work and produce any outputs?
With setNumThreads(0):
import six.moves as sm
import numpy as np
import imgaug as ia
import imgaug.augmenters as iaa
import os
import platform
import cv2

cv2.setNumThreads(0)

print(os.name)
print(platform.system())
print(platform.release())

for iteration, background in enumerate([False, False, False,
                                        True, True, True]):
    image = np.array([[0, 0, 1, 1, 1],
                      [0, 0, 1, 1, 1],
                      [0, 1, 1, 1, 1]], dtype=np.uint8)
    keypoint = ia.Keypoint(x=2, y=1)
    keypoints = [ia.KeypointsOnImage([keypoint], shape=image.shape + (1,))]

    def _lambda_func_images(images, random_state, parents, hooks):
        return images

    def _lambda_func_keypoints(keypoints_on_images, random_state,
                               parents, hooks):
        return keypoints_on_images

    def _assertlambda_func_images(images, random_state, parents, hooks):
        return True

    def _assertlambda_func_keypoints(keypoints_on_images, random_state, parents, hooks):
        return True

    augs = [
        iaa.Sequential([iaa.Fliplr(1.0), iaa.Flipud(1.0)]),
        iaa.SomeOf(1, [iaa.Fliplr(1.0), iaa.Flipud(1.0)]),
        iaa.OneOf([iaa.Fliplr(1.0), iaa.Flipud(1.0)]),
        iaa.Sometimes(1.0, iaa.Fliplr(1)),
        iaa.WithColorspace("HSV", children=iaa.Add((-50, 50))),
        iaa.WithChannels([0], iaa.Add((-50, 50))),
        iaa.Noop(name="Noop-nochange"),
        iaa.Lambda(
            func_images=_lambda_func_images,
            func_keypoints=_lambda_func_keypoints,
            name="Lambda-nochange"
        ),
        iaa.AssertLambda(
            func_images=_assertlambda_func_images,
            func_keypoints=_assertlambda_func_keypoints,
            name="AssertLambda-nochange"
        ),
        iaa.AssertShape(
            (None, 64, 64, 3),
            check_keypoints=False,
            name="AssertShape-nochange"
        ),
        iaa.Resize((0.5, 0.9)),
        iaa.CropAndPad(px=(-50, 50)),
        iaa.Pad(px=(1, 50)),
        iaa.Crop(px=(1, 50)),
        iaa.Fliplr(1.0),
        iaa.Flipud(1.0),
        iaa.Superpixels(p_replace=(0.25, 1.0), n_segments=(16, 128)),
        iaa.ChangeColorspace(to_colorspace="GRAY"),
        iaa.Grayscale(alpha=(0.1, 1.0)),
        iaa.GaussianBlur(1.0),
        iaa.AverageBlur(5),
        iaa.MedianBlur(5),
        iaa.Convolve(np.array([[0, 1, 0],
                               [1, -4, 1],
                               [0, 1, 0]])),
        iaa.Sharpen(alpha=(0.1, 1.0), lightness=(0.8, 1.2)),
        iaa.Emboss(alpha=(0.1, 1.0), strength=(0.8, 1.2)),
        iaa.EdgeDetect(alpha=(0.1, 1.0)),
        iaa.DirectedEdgeDetect(alpha=(0.1, 1.0), direction=(0.0, 1.0)),
        iaa.Add((-50, 50)),
        iaa.AddElementwise((-50, 50)),
        iaa.AdditiveGaussianNoise(scale=(0.1, 1.0)),
        iaa.Multiply((0.6, 1.4)),
        iaa.MultiplyElementwise((0.6, 1.4)),
        iaa.Dropout((0.3, 0.5)),
        iaa.CoarseDropout((0.3, 0.5), size_percent=(0.05, 0.2)),
        iaa.Invert(0.5),
        iaa.Affine(
            scale=(0.7, 1.3),
            translate_percent=(-0.1, 0.1),
            rotate=(-20, 20),
            shear=(-20, 20),
            order=ia.ALL,
            mode=ia.ALL,
            cval=(0, 255)),
        iaa.PiecewiseAffine(scale=(0.1, 0.3)),
        iaa.ElasticTransformation(alpha=0.5)
    ]

    nb_iterations = 100
    image = ia.quokka(size=(64, 64))
    batches = [ia.Batch(images=[np.copy(image)],
                        keypoints=[keypoints[0].deepcopy()])
               for _ in sm.xrange(nb_iterations)]
    for aug in augs:
        print("[#%d bg=%s] Running %s..." % (iteration, background, aug.name))
        nb_changed = 0
        batches_aug = list(aug.augment_batches(batches, background=background))
        for batch_aug in batches_aug:
            image_aug = batch_aug.images_aug[0]
            if (image.shape != image_aug.shape
                    or not np.array_equal(image, image_aug)):
                nb_changed += 1
                if nb_changed > 10:
                    break
        if "-nochange" not in aug.name:
            assert nb_changed > 0
        else:
            assert nb_changed == 0

With set_start_method("spawn") (this may take 5mins or so to finish):
import six.moves as sm
import numpy as np
import imgaug as ia
import imgaug.augmenters as iaa
import os
import platform
import multiprocessing

def main():
    print(os.name)
    print(platform.system())
    print(platform.release())

    multiprocessing.set_start_method("spawn")

    for iteration, background in enumerate([False, False, False,
                                            True, True, True]):
        image = np.array([[0, 0, 1, 1, 1],
                          [0, 0, 1, 1, 1],
                          [0, 1, 1, 1, 1]], dtype=np.uint8)
        keypoint = ia.Keypoint(x=2, y=1)
        keypoints = [ia.KeypointsOnImage([keypoint], shape=image.shape + (1,))]

        augs = [
            iaa.Sequential([iaa.Fliplr(1.0), iaa.Flipud(1.0)]),
            iaa.SomeOf(1, [iaa.Fliplr(1.0), iaa.Flipud(1.0)]),
            iaa.OneOf([iaa.Fliplr(1.0), iaa.Flipud(1.0)]),
            iaa.Sometimes(1.0, iaa.Fliplr(1)),
            iaa.WithColorspace("HSV", children=iaa.Add((-50, 50))),
            iaa.WithChannels([0], iaa.Add((-50, 50))),
            iaa.Noop(name="Noop-nochange"),
            iaa.Lambda(
                func_images=_lambda_func_images,
                func_keypoints=_lambda_func_keypoints,
                name="Lambda-nochange"
            ),
            #iaa.AssertLambda(
            #    func_images=_assertlambda_func_images,
            #    func_keypoints=_assertlambda_func_keypoints,
            #    name="AssertLambda-nochange"
            #),
            #iaa.AssertShape(
            #    (None, 64, 64, 3),
            #    check_keypoints=False,
            #    name="AssertShape-nochange"
            #),
            iaa.Resize((0.5, 0.9)),
            iaa.CropAndPad(px=(-50, 50)),
            iaa.Pad(px=(1, 50)),
            iaa.Crop(px=(1, 50)),
            iaa.Fliplr(1.0),
            iaa.Flipud(1.0),
            iaa.Superpixels(p_replace=(0.25, 1.0), n_segments=(16, 128)),
            iaa.ChangeColorspace(to_colorspace="GRAY"),
            iaa.Grayscale(alpha=(0.1, 1.0)),
            iaa.GaussianBlur(1.0),
            iaa.AverageBlur(5),
            iaa.MedianBlur(5),
            iaa.Convolve(np.array([[0, 1, 0],
                                   [1, -4, 1],
                                   [0, 1, 0]])),
            iaa.Sharpen(alpha=(0.1, 1.0), lightness=(0.8, 1.2)),
            iaa.Emboss(alpha=(0.1, 1.0), strength=(0.8, 1.2)),
            iaa.EdgeDetect(alpha=(0.1, 1.0)),
            iaa.DirectedEdgeDetect(alpha=(0.1, 1.0), direction=(0.0, 1.0)),
            iaa.Add((-50, 50)),
            iaa.AddElementwise((-50, 50)),
            iaa.AdditiveGaussianNoise(scale=(0.1, 1.0)),
            iaa.Multiply((0.6, 1.4)),
            iaa.MultiplyElementwise((0.6, 1.4)),
            iaa.Dropout((0.3, 0.5)),
            iaa.CoarseDropout((0.3, 0.5), size_percent=(0.05, 0.2)),
            iaa.Invert(0.5),
            iaa.Affine(
                scale=(0.7, 1.3),
                translate_percent=(-0.1, 0.1),
                rotate=(-20, 20),
                shear=(-20, 20),
                order=ia.ALL,
                mode=ia.ALL,
                cval=(0, 255)),
            iaa.PiecewiseAffine(scale=(0.1, 0.3)),
            iaa.ElasticTransformation(alpha=0.5)
        ]

        nb_iterations = 100
        image = ia.quokka(size=(64, 64))
        batches = [ia.Batch(images=[np.copy(image)],
                            keypoints=[keypoints[0].deepcopy()])
                   for _ in sm.xrange(nb_iterations)]
        for aug in augs:
            print("[#%d bg=%s] Running %s..." % (iteration, background, aug.name))
            nb_changed = 0
            batches_aug = list(aug.augment_batches(batches, background=background))
            for batch_aug in batches_aug:
                image_aug = batch_aug.images_aug[0]
                if (image.shape != image_aug.shape
                        or not np.array_equal(image, image_aug)):
                    nb_changed += 1
                    if nb_changed > 10:
                        break
            if "-nochange" not in aug.name:
                assert nb_changed > 0
            else:
                assert nb_changed == 0

def _lambda_func_images(images, random_state, parents, hooks):
    return images

def _lambda_func_keypoints(keypoints_on_images, random_state,
                           parents, hooks):
    return keypoints_on_images

def _assertlambda_func_images(images, random_state, parents, hooks):
    return True

def _assertlambda_func_keypoints(keypoints_on_images, random_state, parents, hooks):
    return True

if __name__ == "__main__":
    main()

Thanks @aleju , I will update you after trying the above mentioned changes.

Hi @aleju , Please find attached the output of above two executions (Block1 and Block2).
Block1 was stuck, but Block2 ran without any issues.
block1_debug_output.txt
block2_debug_output.txt
Also i did not get anything after pressing ctrl+c in my environment.
Thanks,
Rakesh

Hi @aleju , Did you a get a chance to look into this? Thanks.

Yes... but as far as I can see, while the second set of tests ran through, it doesn't seem like the library can identify that it is running in NixOS and then conditionally use spawn mode for multiprocessing. The spawn mode seems to be too slow to activate it for all systems (and probably also needs too much memory), so that simple solution isn't an option.
The first set of tests further hang at a different augmenter this time than previously and that augmenter does not use OpenCV but instead scikit-image. It might be that scikit-image uses OpenCV in the background, but I didn't see it when looking through the code. That makes it fairly unlikely that it is an OpenCV issue and more likely that it is a problem or slight behaviour change in the multiprocessing component in NixOS.
Basically I will have to install the OS on an old laptop and see if I can find the issue and fix it in a way that other systems are not affected. That's fairly time intensive and has a high risk of no succeeding, which is why I postponed that so far.

Hi @aleju , Thanks.
If you can suggest the change needed to get this going (what and where), then we can patch it specific for NixOS in our derivations. This will make it usable for us as well.

FYI, we are successfully using imgaug on NixOS environments with no problems. It's just the tests here failing.

FYI, we are successfully using imgaug on NixOS environments with no problems. It's just the tests here failing.

Only the tests involving multiprocessing are failing. If you don't use any multicore augmentation (i.e. augment_batch(..., background=True) or <augmenter>.pool() or imgaug.multicore.Pool), everything should work fine. Even when using these components it might work in real life applications, as the hanging seems to happen unpredictably, so it is unclear what actually causes it.

If you can suggest the change needed to get this going (what and where), then we can patch it specific for NixOS in our derivations. This will make it usable for us as well.

Do you mean patching NixOS or a fork of imgaug? I had hoped that os.name, platform.system() or platform.release() would contain the string NixOS somewhere, so that I can switch to spawn method for that OS. It wouldn't be a great solution as it doesn't really fix the underlying issue but simply avoids it, but at least it would work for now.

@aleju thanks for all the help to the NixOS community! Really appreciate your efforts. Just chiming in that platform.uname() may be a good option:
>>> platform.uname()
uname_result(system='Linux', node='lensman', release='4.19.84', version='#1-NixOS SMP Tue Nov 12 18:21:46 UTC 2019', machine='x86_64', processor='')

Ah, didn't know about platform.uname().version. Thanks for pointing that out.
This adjusted test might work then and shouldn't break anything for other OS:
import six.moves as sm
import numpy as np
import imgaug as ia
import imgaug.augmenters as iaa
import os
import platform
import multiprocessing

def main():
    print(os.name)
    print(platform.system())
    print(platform.release())
    print(platform.version())

    if "NixOS" in platform.version():
        multiprocessing.set_start_method("spawn")

    for iteration, background in enumerate([False, False, False,
                                            True, True, True]):
        image = np.array([[0, 0, 1, 1, 1],
                          [0, 0, 1, 1, 1],
                          [0, 1, 1, 1, 1]], dtype=np.uint8)
        keypoint = ia.Keypoint(x=2, y=1)
        keypoints = [ia.KeypointsOnImage([keypoint], shape=image.shape + (1,))]

        augs = [
            iaa.Sequential([iaa.Fliplr(1.0), iaa.Flipud(1.0)]),
            iaa.SomeOf(1, [iaa.Fliplr(1.0), iaa.Flipud(1.0)]),
            iaa.OneOf([iaa.Fliplr(1.0), iaa.Flipud(1.0)]),
            iaa.Sometimes(1.0, iaa.Fliplr(1)),
            iaa.WithColorspace("HSV", children=iaa.Add((-50, 50))),
            iaa.WithChannels([0], iaa.Add((-50, 50))),
            iaa.Noop(name="Noop-nochange"),
            iaa.Lambda(
                func_images=_lambda_func_images,
                func_keypoints=_lambda_func_keypoints,
                name="Lambda-nochange"
            ),
            #iaa.AssertLambda(
            #    func_images=_assertlambda_func_images,
            #    func_keypoints=_assertlambda_func_keypoints,
            #    name="AssertLambda-nochange"
            #),
            #iaa.AssertShape(
            #    (None, 64, 64, 3),
            #    check_keypoints=False,
            #    name="AssertShape-nochange"
            #),
            iaa.Resize((0.5, 0.9)),
            iaa.CropAndPad(px=(-50, 50)),
            iaa.Pad(px=(1, 50)),
            iaa.Crop(px=(1, 50)),
            iaa.Fliplr(1.0),
            iaa.Flipud(1.0),
            iaa.Superpixels(p_replace=(0.25, 1.0), n_segments=(16, 128)),
            iaa.ChangeColorspace(to_colorspace="GRAY"),
            iaa.Grayscale(alpha=(0.1, 1.0)),
            iaa.GaussianBlur(1.0),
            iaa.AverageBlur(5),
            iaa.MedianBlur(5),
            iaa.Convolve(np.array([[0, 1, 0],
                                   [1, -4, 1],
                                   [0, 1, 0]])),
            iaa.Sharpen(alpha=(0.1, 1.0), lightness=(0.8, 1.2)),
            iaa.Emboss(alpha=(0.1, 1.0), strength=(0.8, 1.2)),
            iaa.EdgeDetect(alpha=(0.1, 1.0)),
            iaa.DirectedEdgeDetect(alpha=(0.1, 1.0), direction=(0.0, 1.0)),
            iaa.Add((-50, 50)),
            iaa.AddElementwise((-50, 50)),
            iaa.AdditiveGaussianNoise(scale=(0.1, 1.0)),
            iaa.Multiply((0.6, 1.4)),
            iaa.MultiplyElementwise((0.6, 1.4)),
            iaa.Dropout((0.3, 0.5)),
            iaa.CoarseDropout((0.3, 0.5), size_percent=(0.05, 0.2)),
            iaa.Invert(0.5),
            iaa.Affine(
                scale=(0.7, 1.3),
                translate_percent=(-0.1, 0.1),
                rotate=(-20, 20),
                shear=(-20, 20),
                order=ia.ALL,
                mode=ia.ALL,
                cval=(0, 255)),
            iaa.PiecewiseAffine(scale=(0.1, 0.3)),
            iaa.ElasticTransformation(alpha=0.5)
        ]

        nb_iterations = 100
        image = ia.quokka(size=(64, 64))
        batches = [ia.Batch(images=[np.copy(image)],
                            keypoints=[keypoints[0].deepcopy()])
                   for _ in sm.xrange(nb_iterations)]
        for aug in augs:
            print("[#%d bg=%s] Running %s..." % (iteration, background, aug.name))
            nb_changed = 0
            batches_aug = list(aug.augment_batches(batches, background=background))
            for batch_aug in batches_aug:
                image_aug = batch_aug.images_aug[0]
                if (image.shape != image_aug.shape
                        or not np.array_equal(image, image_aug)):
                    nb_changed += 1
                    if nb_changed > 10:
                        break
            if "-nochange" not in aug.name:
                assert nb_changed > 0
            else:
                assert nb_changed == 0

def _lambda_func_images(images, random_state, parents, hooks):
    return images

def _lambda_func_keypoints(keypoints_on_images, random_state,
                           parents, hooks):
    return keypoints_on_images

def _assertlambda_func_images(images, random_state, parents, hooks):
    return True

def _assertlambda_func_keypoints(keypoints_on_images, random_state, parents, hooks):
    return True

if __name__ == "__main__":
    main()

Thanks @aleju @tbenst .
I have tried the code snippet shared above. This works fine without any issues on my system.
debug_output_with_NixOS_check.txt
running install tests posix Linux 4.19.80 #1-NixOS SMP Thu Oct 17 20:45:44 UTC 2019 [#0 bg=False] Running UnnamedSequential... [#0 bg=False] Running UnnamedSomeOf... [#0 bg=False] Running UnnamedOneOf... .. .. [#4 bg=True] Running UnnamedElasticTransformation... [#5 bg=True] Running UnnamedSequential... [#5 bg=True] Running UnnamedSomeOf... [#5 bg=True] Running UnnamedOneOf... [#5 bg=True] Running UnnamedSometimes... [#5 bg=True] Running UnnamedWithColorspace... [#5 bg=True] Running UnnamedWithChannels... [#5 bg=True] Running Noop-nochange... [#5 bg=True] Running Lambda-nochange... [#5 bg=True] Running UnnamedResize... [#5 bg=True] Running UnnamedCropAndPad... [#5 bg=True] Running UnnamedPad... [#5 bg=True] Running UnnamedCrop... [#5 bg=True] Running UnnamedFliplr... [#5 bg=True] Running UnnamedFlipud... [#5 bg=True] Running UnnamedSuperpixels... [#5 bg=True] Running UnnamedChangeColorspace... [#5 bg=True] Running UnnamedGrayscale... [#5 bg=True] Running UnnamedGaussianBlur... [#5 bg=True] Running UnnamedAverageBlur... [#5 bg=True] Running UnnamedMedianBlur... [#5 bg=True] Running UnnamedConvolve... [#5 bg=True] Running UnnamedSharpen... [#5 bg=True] Running UnnamedEmboss... [#5 bg=True] Running UnnamedEdgeDetect... [#5 bg=True] Running UnnamedDirectedEdgeDetect... [#5 bg=True] Running UnnamedAdd... [#5 bg=True] Running UnnamedAddElementwise... [#5 bg=True] Running UnnamedAdditiveGaussianNoise... [#5 bg=True] Running UnnamedMultiply... [#5 bg=True] Running UnnamedMultiplyElementwise... [#5 bg=True] Running UnnamedDropout... [#5 bg=True] Running UnnamedCoarseDropout... [#5 bg=True] Running UnnamedInvert... [#5 bg=True] Running UnnamedAffine... [#5 bg=True] Running UnnamedPiecewiseAffine... [#5 bg=True] Running UnnamedElasticTransformation... pythonCatchConflictsPhase pythonRemoveBinBytecodePhase pythonImportsCheckPhase Executing pythonImportsCheckPhase pytestcachePhase

I just merged a patch into master that hopefully fixes the issue. Can you check if the tests run successfully with that version?

I just merged a patch into master that hopefully fixes the issue. Can you check if the tests run successfully with that version?

Thanks. @aleju . I will check and update the outcome.

Thanks @aleju . Tests are executed without any issues on this version.

Thanks @aleju . Tests are executed without any issues on this version.

Hi @aleju , I am again facing this issue. I am debugging on my end, in case I have missed anything. Thanks.

Hi @aleju , sorry for inconvenience. But last time when i reported there was some problem on my branch that caused known behaviour.
The current state is from master:
When the tests started we get this error:
test/test_multicore.py::TestPool::test_augmentations_with_seed_match Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/nix/store/drr8qcgiccfc5by09r5zc30flgwh1mbx-python3-3.7.5/lib/python3.7/multiprocessing/spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "/nix/store/drr8qcgiccfc5by09r5zc30flgwh1mbx-python3-3.7.5/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
  File "/build/imgaug-c268d368f9caeb46d8df4fe73f8f7c988c97dcba/imgaug/multicore.py", line 48, in <module>
    _switch_to_spawn_if_nixos()
  File "/build/imgaug-c268d368f9caeb46d8df4fe73f8f7c988c97dcba/imgaug/multicore.py", line 45, in _switch_to_spawn_if_nixos
    multiprocessing.set_start_method("spawn")
  File "/nix/store/drr8qcgiccfc5by09r5zc30flgwh1mbx-python3-3.7.5/lib/python3.7/multiprocessing/context.py", line 242, in set_start_method
    raise RuntimeError('context has already been set')
RuntimeError: context has already been set

I locally patched this with following two changes:
in **imgaug/multicore.py** replace "multiprocessing.set_start_method(\"spawn\")"  with "multiprocessing.set_start_method(\"spawn\",force=True)"
in **test/test_multicore.py** replace "mock_ssm.assert_called_once_with(\"spawn\")" with mock_ssm.assert_called_once_with(\"spawn\",force=True)"

After this change hang issue has disappeared for augmenter tests.
I have attached the detailed error as attachment img_aug_erros.txt.
 Shall we retest?

Hi @CMCDragonkai, @aleju ,
Tested the changes on my system with https://github.com/aleju/imgaug/pull/526/commits/5c2fa03ee44fb8e4995c46066da4adda9a44f2ef on #526 ,

TestBatchLoader.test_basic_functionality is now Passing.
TestAugmenter_augment_batches.test_augment_batches_with_many_different_augmenters is still failing. error is same as before.

Local patches not needed anymore for:
in **imgaug/multicore.py** replace "multiprocessing.set_start_method(\"spawn\")"  with "multiprocessing.set_start_method(\"spawn\",force=True)"
in **test/test_multicore.py** replace "mock_ssm.assert_called_once_with(\"spawn\")" with mock_ssm.assert_called_once_with(\"spawn\",force=True)"

Thanks.

Sorry, didn't see that there were two new errors.
I just pushed an update to #526 that should fix both issues.
And yes, #526 is supposed to do the same as force=True, just in a bit more elegant way.

Thanks @aleju , finally with this fix imgaug is building on NixOS with all the test case passing.
 I merged the patch to master.

Nixos fix is now merged to upstream. Closing this issue. Thanks again @aleju


GPII-3467: Allow coverage sender to be used without QUnit or Testem.

See GPII-3467 for details.

@amb26, I have addressed your feedback, this should be ready for further review.


UX meeting agenda 3 August 2016

Agenda for tomorrow

Closing dirty split editor: https://github.com/Microsoft/vscode/issues/8983
How to show errors in the editor: https://github.com/Microsoft/vscode/issues/9062
Opening arbitrary commits in diff viewer: https://github.com/Microsoft/vscode/issues/9871
UI for replace all: https://github.com/Microsoft/vscode/issues/9887
Files with the same name: https://github.com/Microsoft/vscode/issues/8499

@Microsoft/vscode - let me know if you want to join the meeting tomorrow.
@sandy081, @joaomoreno - we hope to discuss a couple of issues that have been assigned to you.

We should also discuss designs created for the following items last week...
Merge all 5 vscode panels into 1 #9659
Close icon on terminal/console/etc. pane should be "x": #9638

this is a big list! can we prioritize by what we think we can tackle in Aug milestone and move the rest out for a later discussion?

The full list of UX items in the August milestone is even bigger: https://github.com/Microsoft/vscode/issues?q=is%3Aopen+label%3Aux+milestone%3A"August+2016"
This was my attempt at focusing just on the items I felt we had to discuss.
An alternative could be that we meet more than once to go through the complete list.
I think we could get through the above list in an hour if we manage the discussion, especially if everybody takes a look at the issues before the meeting.

NOTES
How to show errors in the editor: #9062
We will leverage the existing binary file error experience.
Opening arbitrary commits in diff viewer: #9871
We will look holistically at our source control experience to see what is missing, such as Git history.  We don't want to stuff more things into an existing UI piece if would be better in an area we have not yet created.
Merge all 5 vscode panels into 1 #9659
Below is the design we were most interested in pursuing.  It uses a different tab style to avoid users confusing it with editor tabs, which are not interchangeable.
We need to iterate on two more things: 1) How will we show notifications?  2) What is the maximum bounded width of the local command/inputs on the right for each panel so that it won't obscure the tabs or appear strange (biggest offender "Problems" input box)?

Close icon on terminal/console/etc. pane should be "x": #9638
We are leaning towards this design.  It keeps the the commands to add/remove terminals in the context of where they are added/removed.  But we need to to investigate how doable it is implementation-wise, as @isidorn suggested (since these are native boxes).


Confirm showing

[x]  Let admin mark as confirmed

[x]  Prevent further slot-picking

[x]  Notify users confirmed

closed by #27